Get an XCom value in the Airflow on_failure_callback function
This article is a part of my "100 data engineering tutorials in 100 days" challenge. (93/100)
When we want to get an XCom variable in the on_failure_callback, we will face a nasty bug. Using the ti
key to retrieve a value from the context gives us a None value.
1
2
# This does not work
context.get("ti").xcom_pull(key="test")
It happens because in the on_failure_callback the task instance is passed to the function using the task_instance
key. Therefore to get a value from XCom, we must execute this code:
1
2
task: TaskInstance = context.get('task_instance')
task.xcom_pull(key="test")
Alternatively, we can import XCom and access it directly:
1
2
from airflow.models import XCom
XCom.get_one(execution_date = context.get('execution_date'), key='test')
Parsing machine learning logs with Ahana, a managed Presto service, and Cube, a headless BI solution

Check out my article published on the Cube.dev blog!
You may also like
- How to use Airflow backfill to run DAGs for a specified date in the past?
- How to check the next execution date of an Airflow DAG
- Conditionally pick an Airflow DAG branch using an SQL query
- How to add a manual step to an Airflow DAG using the JiraOperator
- What to do when Airflow BashOperator fails with TemplateNotFound error
Bartosz Mikulski
- Data/MLOps engineer by day
- DevRel/copywriter by night
- Python and data engineering trainer
- Conference speaker
- Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
- Twitter: @mikulskibartosz