Get an XCom value in the Airflow on_failure_callback function

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (93/100)

When we want to get an XCom variable in the on_failure_callback, we will face a nasty bug. Using the ti key to retrieve a value from the context gives us a None value.

1
2
# This does not work
context.get("ti").xcom_pull(key="test")

It happens because in the on_failure_callback the task instance is passed to the function using the task_instance key. Therefore to get a value from XCom, we must execute this code:

1
2
task: TaskInstance = context.get('task_instance')
task.xcom_pull(key="test")

Alternatively, we can import XCom and access it directly:

1
2
from airflow.models import XCom
XCom.get_one(execution_date = context.get('execution_date'), key='test')

Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?

Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!

Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.