How to set a different retry delay for every task in an Airflow DAG
When we configure an Airflow DAG, we set the default values of various properties, for example, the retry delay. What should we do when we want to overwrite the delay in an Airflow task? What if we’re going to use a different delay in every task?
In this case, we have to include the retry_delay parameter in the constructor of Airflow operators, which delay we want to change. Every Airflow operator supports this parameter because it is defined in the BaseOperator. For the same reason, it is easy to overlook its existence because Airflow documentation does not show the parameters inherited from base classes.
Would you like to help fight youth unemployment while getting mentoring experience?
Develhope is looking for tutors (part-time, freelancers) for their upcoming Data Engineer Courses.
The role of a tutor is to be the point of contact for students, guiding them throughout the 6-month learning program. The mentor supports learners through 1:1 meetings, giving feedback on assignments, and responding to messages in Discord channels—no live teaching sessions.
Expected availability: 15h/week. You can schedule the 1:1 sessions whenever you want, but the sessions must happen between 9 - 18 (9 am - 6 pm) CEST Monday-Friday.
Check out their job description.
(free advertisement, no affiliate links)
You may also like
- What to do when Airflow BashOperator fails with TemplateNotFound error
- Use HttpSensor to pause an Airflow DAG until a website is available
- How to run Airflow in Docker (with a persistent database)
- How to add an EMR step in Airflow and wait until it finishes running
- How to deal with the jinja2 TemplateNotFound error in Airflow
