How to deal with the jinja2 TemplateNotFound error in Airflow

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (27/100)

Occasionally, when we are making a custom Airflow operator, we may end up in a situation when Airflow complains about not existing jinja template. It may happen when we have an Airflow operator, which calls another operator inside its execute function when both operators have a common template field.

In this situation, Airflow may raise an error saying that: jinja2.exception.TemplateNotFound: content_of_the_field. It is strange because if we have not used a jinja template inside the field value, we may expect it to pass the string without using it as a template. Unfortunately, that does not happen.

Dealing with the issue is quite simple. The wrapping Airflow operator should include the field’s content as a jinja constant and pass it into the inner operator. For example, when our code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = field
        self.extra_args = args
        self.extra_kwargs = kwargs

    def execute(self, context):
        self.another_operator = SomeOtherOperator(field=self.field, *self.extra_args, **self.extra_kwargs)
        self.another_operator.execute(context)


We can fix the jinja template issue like this:

1
2
3
4
5
6
7
8
9
10
11
class SomeOperator(BaseOperator):

    template_fields = ('the_field')

    def __init__(self, field, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.field = "{{ '" + field + "' }}"
        self.extra_args = args
        self.extra_kwargs = kwargs

    ... rest of the code

Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group