How to deal with the jinja2 TemplateNotFound error in Airflow

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (27/100)

Occasionally, when we are making a custom Airflow operator, we may end up in a situation when Airflow complains about not existing jinja template. It may happen when we have an Airflow operator, which calls another operator inside its execute function when both operators have a common template field.

In this situation, Airflow may raise an error saying that: jinja2.exception.TemplateNotFound: content_of_the_field. It is strange because if we have not used a jinja template inside the field value, we may expect it to pass the string without using it as a template. Unfortunately, that does not happen.

Dealing with the issue is quite simple. The wrapping Airflow operator should include the field’s content as a jinja constant and pass it into the inner operator. For example, when our code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = field
        self.extra_args = args
        self.extra_kwargs = kwargs

    def execute(self, context):
        self.another_operator = SomeOtherOperator(field=self.field, *self.extra_args, **self.extra_kwargs)
        self.another_operator.execute(context)

Subscribe to the newsletter and join the free email course.

We can fix the jinja template issue like this:

1
2
3
4
5
6
7
8
9
10
11
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = "{{ '" + field + "' }}"
        self.extra_args = args
        self.extra_kwargs = kwargs

    ... rest of the code

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.

Do you want to work with me at riskmethods?

REMOTE position (available in Poland or Germany)