Use a custom function in Airflow templates

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (47/100)

Adding a custom function to Airflow is quite simple. First, we have do define a function in Python, for example, this one:

1
2
3
def do_something_with_execution_date(execution_date):
    # Imagine that there is some useful code ;)
    ...

When the function is ready, we use the user_defined_macros parameter of the DAG object to pass a dictionary of custom functions:

1
2
3
4
5
6
dag = DAG(
    ...,
    user_defined_macros={
        'custom_function': do_something_with_execution_date,
    }
)

Now, we can use the custom function in any place that supports Airflow templates. Of course, only in the DAGs that have access to the functions.

1
{{ custom_function(execution_date) }};

Note that, I can pass parameters to the function and rename it by using a different name as the dictionary key.


Subscribe to the newsletter and join the free email course.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * data/machine learning engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.