Get an XCom value in the Airflow on_failure_callback function

This article is a part of my "100 data engineering tutorials in 100 days" challenge. (93/100)

When we want to get an XCom variable in the on_failure_callback, we will face a nasty bug. Using the ti key to retrieve a value from the context gives us a None value.

1
2
# This does not work
context.get("ti").xcom_pull(key="test")

It happens because in the on_failure_callback the task instance is passed to the function using the task_instance key. Therefore to get a value from XCom, we must execute this code:

1
2
task: TaskInstance = context.get('task_instance')
task.xcom_pull(key="test")

Alternatively, we can import XCom and access it directly:

1
2
from airflow.models import XCom
XCom.get_one(execution_date = context.get('execution_date'), key='test')


Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.