How to run an Airflow DAG in a loop

Airflow does not support DAGs with loops. After all, the abbreviation DAG stands for Directed Acyclic Graph, so we can’t have cycles. It is also not the standard usage of Airflow, which was built to support daily batch processing.

All of that does not stop us from using a simple trick that lets us run a DAG in a loop. To do that, we have to add a TriggerDagRunOperator as the last task in the DAG. In the task configuration, we specify the DAG id of the DAG that contains the task:

from airflow.operators.dagrun_operator import TriggerDagRunOperator

trigger_self = TriggerDagRunOperator(
    task_id='repeat'
    trigger_dag_id=dag.dag_id,
    dag=dag
)

the_rest_of_the_dag >> trigger_self  # add it as the last task
Older post

How to use xcom_pull to get a variable from another DAG

Get an XCOM variable from another DAG

Newer post

How to check when an Athena table was updated

How to track the time when an Athena table was updated