How to set a different retry delay for every task in an Airflow DAG
When we configure an Airflow DAG, we set the default values of various properties, for example, the retry delay. What should we do when we want to overwrite the delay in an Airflow task? What if we’re going to use a different delay in every task?
In this case, we have to include the
retry_delay parameter in the constructor of Airflow operators, which delay we want to change. Every Airflow operator supports this parameter because it is defined in the
BaseOperator. For the same reason, it is easy to overlook its existence because Airflow documentation does not show the parameters inherited from base classes.
You may also like
- Get an XCom value in the Airflow on_failure_callback function
- How to find the Hive partition closest to a given date
- Dependencies between DAGs: How to wait until another DAG finishes in Airflow?
- How to conditionally skip tasks in an Airflow DAG
- How to run Airflow in Docker (with a persistent database)