How to trigger an Airflow DAG from another DAG
Airflow

How to trigger an Airflow DAG from another DAG

How to trigger another DAG from an Airflow DAG
Bartosz Mikulski
Bartosz Mikulski – 1 min read

I wondered how to use the TriggerDagRunOperator operator since I learned that it exists. I had a few ideas.

There is a concept of SubDAGs in Airflow, so extracting a part of the DAG to another and triggering it using the TriggerDagRunOperator does not look like a correct usage.

The next idea was using it to trigger a compensation action in case of a DAG failure. We can use the BranchPythonOperator to define two code execution paths, choose the first one during regular operation, and the other path in case of an error. In the other branch, we can trigger another DAG using the trigger operator. However, that does not make any sense either. I could put all of the compensation tasks in the other code branch and not bother using the trigger operator and defining a separate DAG.

On the other hand, if I had a few DAGs that require the same compensation actions in case of failures, I could extract the common code to a separate DAG and add only the BranchPythonOperator and the TriggerDagRunOperator to all of the DAGs that must fix something in a case of a failure.

The next idea I had was extracting an expansive computation that does not need to run every time to a separate DAG and trigger it only when necessary. For example, when the input data contains some values.

Still, all of those ideas a little bit exaggerated and overstretched. Perhaps, most of the time, the TriggerDagRunOperator is just overkill.

Usage

The usage of TriggerDagRunOperator is quite simple. All we need is this code:

trigger = TriggerDagRunOperator(
    task_id="trigger_id",
    trigger_dag_id="the_id_of_another_dag",
    dag=dag,
)
Older post

Why does the ExternalTaskSensor get stuck?

How to fix the stuck ExternalTaskSensor

Newer post

Conditionally pick an Airflow DAG branch using an SQL query

How to use the BranchSQLOperator to choose a DAG branch to execute

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020