How to delay an Airflow DAG until a given hour using the DateTimeSensor

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (36/100)

The DateTimeSensor is another part of the Airflow API that seemingly makes no sense. Why would I want to wait until a specified time? For sure, there is little usage of configuration looking like this:

1
2
3
4
5
6
from airflow.sensors.date_time_sensor import DateTimeSensor

time_sensor = DateTimeSensor(
    task_id='wait_until_new_year',
    target_time='2021-01-01T00:00:00',
)

I could imagine only one situation when it is useful. For example, we may have a DAG that will execute only once when we notify the customers that we have just launched a new product. Such mailing will require a few preparation steps, so we can perform those tasks earlier and then wait until the launch date to send the emails.



However, the DateTimeSensor can be used with templates, and that makes it way more powerful. If I have a DAG that must wait until 10 am to execute some tasks, I can configure the sensor using the execution_date parameter:

1
2
3
4
time_sensor = DateTimeSensor(
    task_id='are_we_there_yet',
     target_time='{{ execution_date.replace(hour=10) }}', 
)

What is the advantage of this configuration? When I run a backfill for some past date, it will not get stuck at the time sensor (because the date has already passed). If I used TimeDeltaSensor, it would wait the specified number of minutes even during the backfills, which probably makes no sense.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group


This website DOES NOT use cookies
but you may still see the cookies set earlier if you have already visited it.