How to set a different retry delay for every task in an Airflow DAG
Airflow

How to set a different retry delay for every task in an Airflow DAG

How to use a different retry delay in every Airflow task
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When we configure an Airflow DAG, we set the default values of various properties, for example, the retry delay. What should we do when we want to overwrite the delay in an Airflow task? What if we’re going to use a different delay in every task?

In this case, we have to include the retry_delay parameter in the constructor of Airflow operators, which delay we want to change. Every Airflow operator supports this parameter because it is defined in the BaseOperator. For the same reason, it is easy to overlook its existence because Airflow documentation does not show the parameters inherited from base classes.

Older post

How to find the Hive partition closest to a given date

How to use Airflow to find the Hive partition closest to a given date

Newer post

How to get names of columns with missing values in PySpark

How to get the names of missing properties for every row in a PySpark Dataframe

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020