XCom with return values in Apache Airflow - Commands & Configuration

Introduction

When you run tasks in Airflow, sometimes you want one task to send a small piece of information to another task. XComs let tasks share these little messages easily by returning values.

When a task needs to send a result like a file path or a number to another task.

When you want to pass a status or flag from one task to the next without using external storage.

When you want to chain tasks that depend on each other's output.

When you want to keep your workflow dynamic by using data generated during execution.

When you want to avoid hardcoding values and instead use live results from previous tasks.

Config File - xcom_return_example_dag.py

xcom_return_example_dag.py

from airflow import DAG
from airflow.decorators import task
from airflow.utils.dates import days_ago

with DAG(dag_id='xcom_return_example', start_date=days_ago(1), schedule_interval=None, catchup=False) as dag:

    @task
    def generate_number():
        return 42

    @task
    def print_number(number):
        print(f"Received number: {number}")

    number = generate_number()
    print_number(number)

This DAG defines two tasks using the @task decorator.

generate_number returns the number 42, which Airflow automatically stores as an XCom.

print_number receives the returned value as an argument and prints it.

This shows how returning a value from a task automatically pushes it as an XCom, and downstream tasks can receive it by using the returned variable.

Commands

List all available DAGs to confirm the new DAG is recognized by Airflow.

Terminal

airflow dags list

Expected Output

dag_id           | filepath

xcom_return_example | /path/to/airflow/dags/xcom_return_example_dag.py

Trigger the DAG run manually to execute the tasks and generate XComs.

Terminal

airflow dags trigger xcom_return_example

Expected Output

Created <DagRun xcom_return_example @ 2024-06-01 12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

View logs of the generate_number task to see the returned value and confirm it ran successfully.

Terminal

airflow tasks logs xcom_return_example generate_number --run-id manual__2024-06-01T12:00:00+00:00

Expected Output

[2024-06-01 12:00:01,000] INFO - Task returned value: 42

[2024-06-01 12:00:01,001] INFO - Task succeeded

→

--run-id - Specify the exact DAG run to view logs for

View logs of the print_number task to see the received XCom value printed.

Terminal

airflow tasks logs xcom_return_example print_number --run-id manual__2024-06-01T12:00:00+00:00

Expected Output

Received number: 42

[2024-06-01 12:00:02,000] INFO - Task succeeded

→

--run-id - Specify the exact DAG run to view logs for

Key Concept

If you remember nothing else, remember: returning a value from a task automatically pushes it as an XCom that downstream tasks can receive as input.

Common Mistakes

Not returning the value from the task function and trying to push XCom manually.

Without a return statement, Airflow does not automatically create the XCom, so downstream tasks get no data.

Use a return statement in the task function to send the value as an XCom automatically.

Trying to access XComs directly without passing the returned value as a parameter.

Direct XCom pulls require extra code and are more complex; passing returned values is simpler and cleaner.

Use the returned value from the upstream task as a parameter in the downstream task function.

Summary

Define tasks with @task decorator and return values to automatically create XComs.

Trigger the DAG and check task logs to see returned values and their usage.

Pass the returned value from one task as an argument to another to share data simply.

Practice

1. What does returning a value from a PythonOperator task do in Airflow?

easy

A. Automatically pushes the returned value to XCom for other tasks to use.

B. Saves the value only in the task logs without sharing.

C. Deletes any previous XCom values for that task.

D. Triggers the next task immediately without waiting.

Solution

1. Step 1: Understand PythonOperator return behavior

   When a PythonOperator returns a value, Airflow automatically stores it in XCom for that task instance.

2. Step 2: Recognize XCom purpose

   XCom is designed to share data between tasks, so returning a value pushes it to XCom for downstream tasks.

3. Final Answer:

   Automatically pushes the returned value to XCom for other tasks to use. -> Option A

4. Quick Check:

   Return value = XCom push [OK]

Hint: Return values from PythonOperator go to XCom automatically [OK]

Common Mistakes:

• Thinking return values are only logged, not shared
• Assuming return clears previous XCom data
• Believing return triggers next task immediately

2. Which of the following is the correct way to pull an XCom value in a downstream task?

easy

A. ti.xcom_pull(task_ids='task1', key='return_value')

B. ti.xcom_push(key='result')

C. xcom_pull(task_id='task1')

D. ti.pull_xcom(task_id='task1')

Solution

1. Step 1: Identify correct method signature

   The method to pull XCom data is ti.xcom_pull(), which requires task_ids and optionally a key.

2. Step 2: Match correct parameters

   Correct usage is ti.xcom_pull(task_ids='task1', key='return_value') to get the returned value from task1.

3. Final Answer:

   ti.xcom_pull(task_ids='task1', key='return_value') -> Option A

4. Quick Check:

   Use ti.xcom_pull with task_ids and key [OK]

Hint: Use ti.xcom_pull(task_ids, key) to get XCom data [OK]

Common Mistakes:

• Using xcom_push instead of xcom_pull to get data
• Wrong parameter names like task_id instead of task_ids
• Calling non-existent methods like ti.pull_xcom

3. Given this DAG snippet:

def task1():
    return {'value': 42}

def task2(ti):
    data = ti.xcom_pull(task_ids='task1')
    print(data)

What will task2 print when executed?

medium

A. None

B. {'return_value': {'value': 42}}

C. 42

D. Error: xcom_pull missing key

Solution

1. Step 1: Understand task1 return value

   task1 returns a dictionary {'value': 42}, which is pushed to XCom automatically.

2. Step 2: Analyze task2 xcom_pull call

   task2 calls ti.xcom_pull(task_ids='task1') without a key, so it pulls all XCom values from task1 as a dictionary {'return_value': {'value': 42}}.

3. Final Answer:

   {'return_value': {'value': 42}} -> Option B

4. Quick Check:

   Pull without key returns XComs dict [OK]

Hint: Pull without key returns XComs dict [OK]

Common Mistakes:

• Expecting only the inner value without specifying key
• Assuming None if key is not given
• Thinking it raises an error without key

4. What is wrong with this code snippet?

def task1():
    return 100

def task2(ti):
    value = ti.xcom_pull(task_ids='task1', key='value')
    print(value)

medium

A. task2 must use task_id instead of task_ids in xcom_pull.

B. task1 should push to XCom manually using xcom_push.

C. task2 pulls with wrong key; default return key is 'return_value'.

D. task1 return value must be a string, not int.

Solution

1. Step 1: Check task1 return and XCom key

   task1 returns 100, which is stored in XCom with the default key 'return_value'.

2. Step 2: Analyze task2 xcom_pull key usage

   task2 tries to pull with key='value', which does not exist, so it will return None.

3. Final Answer:

   task2 pulls with wrong key; default return key is 'return_value'. -> Option C

4. Quick Check:

   Default XCom key for return is 'return_value' [OK]

Hint: Default XCom key for return is 'return_value' [OK]

Common Mistakes:

• Thinking return requires manual xcom_push
• Using wrong key name in xcom_pull
• Confusing task_id and task_ids parameter names

5. You want to pass a dictionary from task1 to task3 through task2 without modifying task2. Which is the best way to do this using XCom return values?

hard

A. Use Airflow Variables instead of XCom for this data passing.

B. Have task1 push manually; task2 returns the same dictionary to pass it on.

C. Modify task2 to pull from task1 and push to task3 manually.

D. Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1').

Solution

1. Step 1: Understand XCom chaining

   XCom stores data per task instance; tasks can pull data from any previous task by specifying task_ids.

2. Step 2: Avoid modifying task2

   Since task2 should not be changed, task3 can pull data directly from task1's XCom return value.

3. Final Answer:

   Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1'). -> Option D

4. Quick Check:

   Pull directly from original task to avoid modifying intermediates [OK]

Hint: Pull XCom directly from original task to skip intermediate tasks [OK]

Common Mistakes:

• Thinking data must pass through every intermediate task
• Modifying tasks unnecessarily to forward XCom
• Using Variables instead of XCom for task-to-task data