How to Use xcom_push in Airflow: Simple Guide

In Airflow, use xcom_push inside a task to send data to other tasks by specifying a key and value. This allows tasks to share information dynamically during workflow execution.

Syntax

The xcom_push method is called inside a task's Python function or operator to send data. It requires a key to identify the data and a value which is the data to share.

Typical usage:

• key: A string to name the data.
• value: Any serializable Python object to share.

python

task_instance.xcom_push(key='my_key', value='my_value')

Example

This example shows two tasks: one pushes a value using xcom_push, and the other pulls it using xcom_pull. It demonstrates how to share data between tasks in the same DAG.

python

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def push_function(ti):
    ti.xcom_push(key='sample_key', value='Hello from push task')

def pull_function(ti):
    pulled_value = ti.xcom_pull(key='sample_key', task_ids='push_task')
    print(f'Pulled value: {pulled_value}')

default_args = {'start_date': datetime(2024, 1, 1)}

dag = DAG('xcom_push_example', default_args=default_args, schedule_interval=None)

push_task = PythonOperator(
    task_id='push_task',
    python_callable=push_function,
    dag=dag
)

pull_task = PythonOperator(
    task_id='pull_task',
    python_callable=pull_function,
    dag=dag
)

push_task >> pull_task

Output

Pulled value: Hello from push task

Common Pitfalls

Common mistakes when using xcom_push include:

• Not specifying a key, which defaults to return_value and can cause confusion.
• Trying to push non-serializable objects, which causes errors.
• Forgetting to pull the XCom with the correct key and task_id.
• Using xcom_push outside of a task context (like in DAG definition), which won't work.

Example of wrong and right usage:

python

# Wrong: pushing outside task context
# ti.xcom_push(key='key', value='value')  # ti undefined here

# Right: inside task function
def task_func(ti):
    ti.xcom_push(key='key', value='value')

Quick Reference

Method	Description	Parameters
xcom_push	Send data from a task	key (str), value (any serializable)
xcom_pull	Retrieve data in a task	key (str), task_ids (str or list)
Default key	If no key given, uses 'return_value'	N/A
Context	Must be called inside task function	Access via task instance (ti)

Key Takeaways

Use xcom_push inside task functions to share data with a key and value.

Always push serializable data and specify a clear key to avoid confusion.

Pull data with xcom_pull using the correct key and task ID.

Do not call xcom_push outside of task execution context.

xcom_push enables dynamic communication between Airflow tasks.