Pushing and pulling XCom values in Apache Airflow - Step-by-Step Execution

Process Flow - Pushing and pulling XCom values

Task A runs

↓

Push XCom value

↓

Task B waits

↓

Pull XCom value

↓

Task B uses value

This flow shows how one task pushes a value to XCom and another task pulls it later to use.

Execution Sample

Apache Airflow

def task_a(ti):
    ti.xcom_push(key='my_key', value='hello')

def task_b(ti):
    val = ti.xcom_pull(key='my_key', task_ids='task_a')
    print(val)

Task A pushes 'hello' to XCom with key 'my_key'. Task B pulls it and prints.

Process Table

Step	Action	XCom State	Output
1	Task A starts	{}
2	Task A pushes key='my_key', value='hello'	{"task_a": {"my_key": "hello"}}
3	Task A finishes	{"task_a": {"my_key": "hello"}}
4	Task B starts	{"task_a": {"my_key": "hello"}}
5	Task B pulls key='my_key' from task_a	{"task_a": {"my_key": "hello"}}	hello
6	Task B prints pulled value	{"task_a": {"my_key": "hello"}}	hello
7	Task B finishes	{"task_a": {"my_key": "hello"}}

💡 Task B finishes after pulling and printing the XCom value.

Status Tracker

Variable	Start	After Step 2	After Step 5	Final
XCom	{}	{"task_a": {"my_key": "hello"}}	{"task_a": {"my_key": "hello"}}	{"task_a": {"my_key": "hello"}}
val (in task_b)	None	None	'hello'	'hello'

Key Moments - 2 Insights

Why does Task B need to specify task_ids='task_a' when pulling?

Because XCom values are stored per task, Task B must specify which task's XCom to pull. See execution_table step 5 where task_b pulls from task_a.

What happens if Task A never pushes an XCom value?

Task B's pull will return None or empty, so the variable 'val' will be None. This is shown by the initial state in variable_tracker before step 5.

Visual Quiz - 3 Questions

Look at the execution table at step 2, what is the XCom state?

A {"task_b": {"my_key": "hello"}}

B {}

C {"task_a": {"my_key": "hello"}}

D None

💡 Hint

Check the 'XCom State' column at step 2 in the execution_table.

At which step does Task B get the value 'hello' from XCom?

A Step 5

B Step 3

C Step 4

D Step 7

💡 Hint

Look at the 'Output' column in execution_table where 'hello' appears.

If Task A pushed value 'world' instead of 'hello', what would Task B print?

A hello

B world

C None

D Error

💡 Hint

Refer to variable_tracker for 'val' after step 5 and imagine the pushed value changed.

Concept Snapshot

XCom lets tasks share small data.
Task pushes with ti.xcom_push(key, value).
Other tasks pull with ti.xcom_pull(key, task_ids).
Pull needs task_ids to find the right task's data.
XCom values persist between tasks in the same DAG run.

Full Transcript

In Airflow, tasks can share data using XComs. One task pushes a value with a key. Another task pulls that value by specifying the key and the task that pushed it. This example shows Task A pushing 'hello' with key 'my_key'. Task B pulls this value and prints it. The execution table traces each step: Task A starts, pushes the value, finishes; Task B starts, pulls the value, prints it, and finishes. The variable tracker shows how the XCom dictionary and the pulled variable 'val' change over time. Key points include the need to specify the task_id when pulling and what happens if no value is pushed. The quiz tests understanding of these steps and the data flow.