How to deal with the jinja2 TemplateNotFound error in Airflow

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (27/100)

Occasionally, when we are making a custom Airflow operator, we may end up in a situation when Airflow complains about not existing jinja template. It may happen when we have an Airflow operator, which calls another operator inside its execute function when both operators have a common template field.

In this situation, Airflow may raise an error saying that: jinja2.exception.TemplateNotFound: content_of_the_field. It is strange because if we have not used a jinja template inside the field value, we may expect it to pass the string without using it as a template. Unfortunately, that does not happen.

Dealing with the issue is quite simple. The wrapping Airflow operator should include the field’s content as a jinja constant and pass it into the inner operator. For example, when our code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = field
        self.extra_args = args
        self.extra_kwargs = kwargs

    def execute(self, context):
        self.another_operator = SomeOtherOperator(field=self.field, *self.extra_args, **self.extra_kwargs)
        self.another_operator.execute(context)

We can fix the jinja template issue like this:

1
2
3
4
5
6
7
8
9
10
11
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = "{{ '" + field + "' }}"
        self.extra_args = args
        self.extra_kwargs = kwargs

    ... rest of the code

Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?

Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!

Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.