How to set a different retry delay for every task in an Airflow DAG
When we configure an Airflow DAG, we set the default values of various properties, for example, the retry delay. What should we do when we want to overwrite the delay in an Airflow task? What if we’re going to use a different delay in every task?
In this case, we have to include the retry_delay parameter in the constructor of Airflow operators, which delay we want to change. Every Airflow operator supports this parameter because it is defined in the BaseOperator. For the same reason, it is easy to overlook its existence because Airflow documentation does not show the parameters inherited from base classes.
You may also like
- How to check the next execution date of an Airflow DAG
- Send SMS from an Airflow DAG using AWS SNS
- How to delay an Airflow DAG until a given hour using the DateTimeSensor
- Use HttpSensor to pause an Airflow DAG until a website is available
- How to add a manual step to an Airflow DAG using the JiraOperator
Remember to share on social media! If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.
If you want to contact me, send me a message on LinkedIn or Twitter.
Would you like to have a call and talk? Please schedule a meeting using this link.
