How to run an Airflow DAG in a loop

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (63/100)

Airflow does not support DAGs with loops. After all, the abbreviation DAG stands for Directed Acyclic Graph, so we can’t have cycles. It is also not the standard usage of Airflow, which was built to support daily batch processing.

All of that does not stop us from using a simple trick that lets us run a DAG in a loop. To do that, we have to add a TriggerDagRunOperator as the last task in the DAG. In the task configuration, we specify the DAG id of the DAG that contains the task:

1
2
3
4
5
6
7
8
9
from airflow.operators.dagrun_operator import TriggerDagRunOperator

trigger_self = TriggerDagRunOperator(
    task_id='repeat'
    trigger_dag_id=dag.dag_id,
    dag=dag
)

the_rest_of_the_dag >> trigger_self  # add it as the last task

Would you like to help fight youth unemployment while getting mentoring experience?

Develhope is looking for tutors (part-time, freelancers) for their upcoming Data Engineer Courses.

The role of a tutor is to be the point of contact for students, guiding them throughout the 6-month learning program. The mentor supports learners through 1:1 meetings, giving feedback on assignments, and responding to messages in Discord channels—no live teaching sessions.

Expected availability: 15h/week. You can schedule the 1:1 sessions whenever you want, but the sessions must happen between 9 - 18 (9 am - 6 pm) CEST Monday-Friday.

Check out their job description.

(free advertisement, no affiliate links)


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.


Bartosz Mikulski
Bartosz Mikulski * MLOps Engineer / data engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.