XCom with return values in Apache Airflow - Step-by-Step Execution

Process Flow - XCom with return values

Task A runs

↓

Task A returns value

↓

Airflow stores value in XCom

↓

Task B starts

↓

Task B pulls value from XCom

↓

Task B uses value

Task A runs and returns a value, which Airflow saves in XCom. Task B then pulls this value from XCom to use it.

Execution Sample

Apache Airflow

def task_a():
    return 'hello from A'

def task_b(ti):
    msg = ti.xcom_pull(task_ids='task_a')
    print(msg)

Task A returns a string; Task B pulls that string from XCom and prints it.

Process Table

Step	Action	Task	XCom State	Output/Result
1	Run task_a()	task_a	XCom empty	task_a returns 'hello from A'
2	Store return value in XCom	task_a	XCom stores {'task_a': 'hello from A'}	Value saved
3	Start task_b()	task_b	XCom has {'task_a': 'hello from A'}	task_b starts
4	Pull value from XCom	task_b	XCom unchanged	Pulled 'hello from A'
5	Print pulled value	task_b	XCom unchanged	Output: hello from A
6	task_b ends	task_b	XCom unchanged	Execution complete

💡 task_b finishes after pulling and printing the value from XCom

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 4	Final
XCom	empty	empty	{'task_a': 'hello from A'}	{'task_a': 'hello from A'}	{'task_a': 'hello from A'}
msg (in task_b)	undefined	undefined	undefined	'hello from A'	'hello from A'

Key Moments - 2 Insights

Why does task_b need to pull the value from XCom instead of receiving it directly?

Because Airflow tasks run independently, the return value from task_a is stored in XCom. task_b must explicitly pull it from XCom, as shown in step 4 of the execution_table.

What happens if task_a does not return anything?

If task_a returns nothing, XCom stores None or no value for task_a. Then, when task_b pulls from XCom (step 4), it gets None, so no useful data is passed.

Visual Quiz - 3 Questions

Look at the execution_table at step 2. What is stored in XCom?

A The print output from task_b

B An empty dictionary

C The string 'hello from A' under task_a key

D Nothing is stored yet

💡 Hint

Check the 'XCom State' column at step 2 in execution_table

At which step does task_b retrieve the value from XCom?

A Step 4

B Step 1

C Step 3

D Step 5

💡 Hint

Look for 'Pull value from XCom' action in execution_table

If task_a returned None, what would 'msg' be after step 4?

A 'hello from A'

B None

C An error

D Empty string

💡 Hint

Refer to variable_tracker for 'msg' values and key_moments explanation about no return

Concept Snapshot

XCom lets Airflow tasks share data.
Task returns a value, Airflow stores it in XCom.
Other tasks pull this value by task_id.
Use ti.xcom_pull(task_ids='task_id') to get it.
Useful for passing small data between tasks.

Full Transcript

In Airflow, tasks run separately. When a task returns a value, Airflow saves it in a special place called XCom. Another task can then ask XCom for that value using the task's ID. This way, tasks can share information even though they run independently. For example, task_a returns 'hello from A', Airflow stores it, and task_b pulls and prints it. If task_a returns nothing, task_b gets None. This process helps tasks communicate small pieces of data.

Practice

1. What does returning a value from a PythonOperator task do in Airflow?

easy

A. Automatically pushes the returned value to XCom for other tasks to use.

B. Saves the value only in the task logs without sharing.

C. Deletes any previous XCom values for that task.

D. Triggers the next task immediately without waiting.

Solution

1. Step 1: Understand PythonOperator return behavior

   When a PythonOperator returns a value, Airflow automatically stores it in XCom for that task instance.

2. Step 2: Recognize XCom purpose

   XCom is designed to share data between tasks, so returning a value pushes it to XCom for downstream tasks.

3. Final Answer:

   Automatically pushes the returned value to XCom for other tasks to use. -> Option A

4. Quick Check:

   Return value = XCom push [OK]

Hint: Return values from PythonOperator go to XCom automatically [OK]

Common Mistakes:

• Thinking return values are only logged, not shared
• Assuming return clears previous XCom data
• Believing return triggers next task immediately

2. Which of the following is the correct way to pull an XCom value in a downstream task?

easy

A. ti.xcom_pull(task_ids='task1', key='return_value')

B. ti.xcom_push(key='result')

C. xcom_pull(task_id='task1')

D. ti.pull_xcom(task_id='task1')

Solution

1. Step 1: Identify correct method signature

   The method to pull XCom data is ti.xcom_pull(), which requires task_ids and optionally a key.

2. Step 2: Match correct parameters

   Correct usage is ti.xcom_pull(task_ids='task1', key='return_value') to get the returned value from task1.

3. Final Answer:

   ti.xcom_pull(task_ids='task1', key='return_value') -> Option A

4. Quick Check:

   Use ti.xcom_pull with task_ids and key [OK]

Hint: Use ti.xcom_pull(task_ids, key) to get XCom data [OK]

Common Mistakes:

• Using xcom_push instead of xcom_pull to get data
• Wrong parameter names like task_id instead of task_ids
• Calling non-existent methods like ti.pull_xcom

3. Given this DAG snippet:

def task1():
    return {'value': 42}

def task2(ti):
    data = ti.xcom_pull(task_ids='task1')
    print(data)

What will task2 print when executed?

medium

A. None

B. {'return_value': {'value': 42}}

C. 42

D. Error: xcom_pull missing key

Solution

1. Step 1: Understand task1 return value

   task1 returns a dictionary {'value': 42}, which is pushed to XCom automatically.

2. Step 2: Analyze task2 xcom_pull call

   task2 calls ti.xcom_pull(task_ids='task1') without a key, so it pulls all XCom values from task1 as a dictionary {'return_value': {'value': 42}}.

3. Final Answer:

   {'return_value': {'value': 42}} -> Option B

4. Quick Check:

   Pull without key returns XComs dict [OK]

Hint: Pull without key returns XComs dict [OK]

Common Mistakes:

• Expecting only the inner value without specifying key
• Assuming None if key is not given
• Thinking it raises an error without key

4. What is wrong with this code snippet?

def task1():
    return 100

def task2(ti):
    value = ti.xcom_pull(task_ids='task1', key='value')
    print(value)

medium

A. task2 must use task_id instead of task_ids in xcom_pull.

B. task1 should push to XCom manually using xcom_push.

C. task2 pulls with wrong key; default return key is 'return_value'.

D. task1 return value must be a string, not int.

Solution

1. Step 1: Check task1 return and XCom key

   task1 returns 100, which is stored in XCom with the default key 'return_value'.

2. Step 2: Analyze task2 xcom_pull key usage

   task2 tries to pull with key='value', which does not exist, so it will return None.

3. Final Answer:

   task2 pulls with wrong key; default return key is 'return_value'. -> Option C

4. Quick Check:

   Default XCom key for return is 'return_value' [OK]

Hint: Default XCom key for return is 'return_value' [OK]

Common Mistakes:

• Thinking return requires manual xcom_push
• Using wrong key name in xcom_pull
• Confusing task_id and task_ids parameter names

5. You want to pass a dictionary from task1 to task3 through task2 without modifying task2. Which is the best way to do this using XCom return values?

hard

A. Use Airflow Variables instead of XCom for this data passing.

B. Have task1 push manually; task2 returns the same dictionary to pass it on.

C. Modify task2 to pull from task1 and push to task3 manually.

D. Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1').

Solution

1. Step 1: Understand XCom chaining

   XCom stores data per task instance; tasks can pull data from any previous task by specifying task_ids.

2. Step 2: Avoid modifying task2

   Since task2 should not be changed, task3 can pull data directly from task1's XCom return value.

3. Final Answer:

   Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1'). -> Option D

4. Quick Check:

   Pull directly from original task to avoid modifying intermediates [OK]

Hint: Pull XCom directly from original task to skip intermediate tasks [OK]

Common Mistakes:

• Thinking data must pass through every intermediate task
• Modifying tasks unnecessarily to forward XCom
• Using Variables instead of XCom for task-to-task data