How to render an Airflow template for testing
Do you know what I do when I have to test a failing Airflow task? I copy the rendered template and run the code on my own, without Airflow. It is way easier to fiddle with the parameters, for example, to get more verbose output.
To get a rendered template, I used to open an instance of the task, copy the code, and change the parameters to match the execution date I wanted. That is the long and error-prone way to do it.
It turns out that the Airflow command-line interface has a command that generates a rendered template of a given task for the execution date we choose. No more copy-pasting and looking for parameters to change!
All we need is this command:
1 airflow render dag_id task_id execution_date
Did you enjoy reading this article?
Would you like to learn more about leveraging AI to drive growth and innovation, software craft in data engineering, and MLOps?
Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!
You may also like
- How to deal with the jinja2 TemplateNotFound error in Airflow
- How to use Virtualenv to prepare a separate environment for Python function running in Airflow
- What to do when Airflow BashOperator fails with TemplateNotFound error
- Conditionally pick an Airflow DAG branch using an SQL query
- How to use AWSAthenaOperator in Airflow to verify that a DAG finished successfully
- MLOps engineer by day
- AI and data engineering consultant by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz
- Mastodon: @firstname.lastname@example.org