Get the date of the previous successful DAG run in Airflow.

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (70/100)

Airflow macros have three variables that we can access to get the dates related to the previous DAG runs.

First, we can access the previous execution date by using the {{ prev_execution_date }} macro. In my opinion, it has limited usage because, in general, we know how often the DAG runs and what should be the previous execution date.

A way more useful variable is the {{ prev_execution_date_success }} , which gives us the previous successful DAG run’s execution date. We can use it to filter the data and retrieve only the values that were not available during the last successful run.

If we use the current date as a filter while retrieving the data, we may also need the {{ prev_start_date_success }} variable, which returns the start time of the previous successful run.

Subscribe to the newsletter and join the free email course.

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.

Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.