How to delay an Airflow DAG until a given hour using the DateTimeSensor
The DateTimeSensor is another part of the Airflow API that seemingly makes no sense. Why would I want to wait until a specified time? For sure, there is little usage of configuration looking like this:
1
2
3
4
5
6
from airflow.sensors.date_time_sensor import DateTimeSensor
time_sensor = DateTimeSensor(
task_id='wait_until_new_year',
target_time='2021-01-01T00:00:00',
)
I could imagine only one situation when it is useful. For example, we may have a DAG that will execute only once when we notify the customers that we have just launched a new product. Such mailing will require a few preparation steps, so we can perform those tasks earlier and then wait until the launch date to send the emails.
However, the DateTimeSensor can be used with templates, and that makes it way more powerful. If I have a DAG that must wait until 10 am to execute some tasks, I can configure the sensor using the execution_date
parameter:
1
2
3
4
time_sensor = DateTimeSensor(
task_id='are_we_there_yet',
target_time='{{ execution_date.replace(hour=10) }}',
)
What is the advantage of this configuration? When I run a backfill for some past date, it will not get stuck at the time sensor (because the date has already passed). If I used TimeDeltaSensor, it would wait the specified number of minutes even during the backfills, which probably makes no sense.
Did you enjoy reading this article?
Would you like to learn more about leveraging AI to drive growth and innovation, software craft in data engineering, and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like
- Use a custom function in Airflow templates
- What to do when Airflow BashOperator fails with TemplateNotFound error
- How to use Virtualenv to prepare a separate environment for Python function running in Airflow
- Conditionally pick an Airflow DAG branch using an SQL query
- How to retrieve the statuses of the recent DAG executions from Airflow database

Bartosz Mikulski
- MLOps engineer by day
- AI and data engineering consultant by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz
- Mastodon: @mikulskibartosz@mathstodon.xyz