Get an XCom value in the Airflow on_failure_callback function
Airflow

Get an XCom value in the Airflow on_failure_callback function

How to get the task instance in the on_failure_callback to get access to XCom
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When we want to get an XCom variable in the on_failure_callback, we will face a nasty bug. Using the ti key to retrieve a value from the context gives us a None value.

# This does not work
context.get("ti").xcom_pull(key="test")

It happens because in the on_failure_callback the task instance is passed to the function using the task_instance key. Therefore to get a value from XCom, we must execute this code:

task: TaskInstance = context.get('task_instance')
task.xcom_pull(key="test")

Alternatively, we can import XCom and access it directly:

from airflow.models import XCom
XCom.get_one(execution_date = context.get('execution_date'), key='test')
Older post

Add the row insertion time to a MySQL table

Automatically add the insertion and update time in MySQL

Newer post

How to get a notification when a new file is uploaded to an S3 bucket

Get a Slack notification when a file is uploaded to an S3 bucket

Related Posts

Airflow Prophet

Anomaly detection in Airflow DAG using Prophet library

How to detect problems in Airflow pipeline using Prophet for time series anomaly detection
Published on: 12 Feb 2021

Airflow

How to restart a stuck Airflow DAG

What to do when an Airflow DAG gets stuck and does not want to run
Published on: 31 Dec 2020

Airflow

Why does the DayOfWeekSensor exist in Airflow?

How to make an Airflow DAG wait until a specified day of the week
Published on: 30 Dec 2020