Get the date of the previous successful DAG run in Airflow.
Airflow

Get the date of the previous successful DAG run in Airflow.

Get the start time or the execution date of the previous successful DAG run in Airflow
Bartosz Mikulski
Bartosz Mikulski – 1 min read

Airflow macros have three variables that we can access to get the dates related to the previous DAG runs.

First, we can access the previous execution date by using the {{ prev_execution_date }} macro. In my opinion, it has limited usage because, in general, we know how often the DAG runs and what should be the previous execution date.

A way more useful variable is the {{ prev_execution_date_success }} , which gives us the previous successful DAG run’s execution date. We can use it to filter the data and retrieve only the values that were not available during the last successful run.

If we use the current date as a filter while retrieving the data, we may also need the {{ prev_start_date_success }} variable, which returns the start time of the previous successful run.

Older post

How to prevent Airflow from backfilling old DAG runs

How to disable backfilling of an Airflow DAG or skip a part of the DAG during a backfill

Newer post

How to find the Hive partition closest to a given date

How to use Airflow to find the Hive partition closest to a given date

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020