How to run an Airflow DAG in a loop
Airflow does not support DAGs with loops. After all, the abbreviation DAG stands for Directed Acyclic Graph, so we can’t have cycles. It is also not the standard usage of Airflow, which was built to support daily batch processing.
All of that does not stop us from using a simple trick that lets us run a DAG in a loop. To do that, we have to add a
TriggerDagRunOperator as the last task in the DAG. In the task configuration, we specify the DAG id of the DAG that contains the task:
1 2 3 4 5 6 7 8 9 from airflow.operators.dagrun_operator import TriggerDagRunOperator trigger_self = TriggerDagRunOperator( task_id='repeat' trigger_dag_id=dag.dag_id, dag=dag ) the_rest_of_the_dag >> trigger_self # add it as the last task
Did you enjoy reading this article?
Would you like to learn more about leveraging AI to drive growth and innovation, software craft in data engineering, and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like
- How to run Airflow in Docker (with a persistent database)
- Conditionally pick an Airflow DAG branch using an SQL query
- How to restart a stuck Airflow DAG
- How to add an EMR step in Airflow and wait until it finishes running
- How to use AWSAthenaOperator in Airflow to verify that a DAG finished successfully
- MLOps engineer by day
- AI and data engineering consultant by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz
- Mastodon: @firstname.lastname@example.org