How to trigger an Airflow DAG from another DAG
I wondered how to use the
TriggerDagRunOperator operator since I learned that it exists. I had a few ideas.
There is a concept of SubDAGs in Airflow, so extracting a part of the DAG to another and triggering it using the
TriggerDagRunOperator does not look like a correct usage.
The next idea was using it to trigger a compensation action in case of a DAG failure. We can use the
BranchPythonOperator to define two code execution paths, choose the first one during regular operation, and the other path in case of an error. In the other branch, we can trigger another DAG using the trigger operator. However, that does not make any sense either. I could put all of the compensation tasks in the other code branch and not bother using the trigger operator and defining a separate DAG.
On the other hand, if I had a few DAGs that require the same compensation actions in case of failures, I could extract the common code to a separate DAG and add only the
BranchPythonOperator and the
TriggerDagRunOperator to all of the DAGs that must fix something in a case of a failure.
The next idea I had was extracting an expansive computation that does not need to run every time to a separate DAG and trigger it only when necessary. For example, when the input data contains some values.
Still, all of those ideas a little bit exaggerated and overstretched. Perhaps, most of the time, the
TriggerDagRunOperator is just overkill.
The usage of
TriggerDagRunOperator is quite simple. All we need is this code:
1 2 3 4 5 trigger = TriggerDagRunOperator( task_id="trigger_id", trigger_dag_id="the_id_of_another_dag", dag=dag, )
Did you enjoy reading this article?
Would you like to learn more about leveraging AI to drive growth and innovation, software craft in data engineering, and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like
- How to prevent Airflow from backfilling old DAG runs
- How to retrieve the statuses of the recent DAG executions from Airflow database
- How to check whether a YARN application has finished
- Run a command on a remote server using SSH in Airflow
- What to do when Airflow BashOperator fails with TemplateNotFound error
- MLOps engineer by day
- AI and data engineering consultant by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz
- Mastodon: @email@example.com