How to run an Airflow DAG in a loop
Airflow does not support DAGs with loops. After all, the abbreviation DAG stands for Directed Acyclic Graph, so we can’t have cycles. It is also not the standard usage of Airflow, which was built to support daily batch processing.
All of that does not stop us from using a simple trick that lets us run a DAG in a loop. To do that, we have to add a TriggerDagRunOperator as the last task in the DAG. In the task configuration, we specify the DAG id of the DAG that contains the task:
1
2
3
4
5
6
7
8
9
from airflow.operators.dagrun_operator import TriggerDagRunOperator
trigger_self = TriggerDagRunOperator(
task_id='repeat'
trigger_dag_id=dag.dag_id,
dag=dag
)
the_rest_of_the_dag >> trigger_self # add it as the last task
Would you like to help fight youth unemployment while getting mentoring experience?
Develhope is looking for tutors (part-time, freelancers) for their upcoming Data Engineer Courses.
The role of a tutor is to be the point of contact for students, guiding them throughout the 6-month learning program. The mentor supports learners through 1:1 meetings, giving feedback on assignments, and responding to messages in Discord channels—no live teaching sessions.
Expected availability: 15h/week. You can schedule the 1:1 sessions whenever you want, but the sessions must happen between 9 - 18 (9 am - 6 pm) CEST Monday-Friday.
Check out their job description.
(free advertisement, no affiliate links)
You may also like
- How to use Virtualenv to prepare a separate environment for Python function running in Airflow
- How to retrieve the statuses of the recent DAG executions from Airflow database
- How to add an EMR step in Airflow and wait until it finishes running
- How to prevent Airflow from backfilling old DAG runs
- Dependencies between DAGs: How to wait until another DAG finishes in Airflow?
