XCom with return values in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

XCom Push and Pull Output

Given the following Airflow task code, what will be the output of the XCom pull in the downstream task?

Apache Airflow

def push_function(ti):
    ti.xcom_push(key='result', value=42)

def pull_function(ti):
    value = ti.xcom_pull(key='result', task_ids='push_task')
    print(value)

# Assume push_function runs as 'push_task' and pull_function runs as 'pull_task' downstream.

A 0

B None

C KeyError

D 42

💡 Hint

Remember that xcom_push stores a value with a key, and xcom_pull retrieves it using the same key and task id.

The push_function stores the integer 42 with key 'result'. The pull_function retrieves it using the same key and task id, so it prints 42.

🧠 Conceptual

intermediate

XCom Return Value Behavior

What happens when a PythonOperator task returns a value in Airflow?

A The return value is ignored and not stored anywhere.

B The return value is automatically pushed to XCom with key 'return_value'.

C The return value causes the task to fail.

D The return value is stored in a log file only.

💡 Hint

Think about how Airflow captures outputs from PythonOperator tasks.

PythonOperator automatically pushes the return value of the Python callable to XCom with the key 'return_value'. This allows downstream tasks to pull it easily.

❓ Troubleshoot

advanced

XCom Pull Returns None

You have two tasks: 'task_a' pushes a value to XCom with key 'data'. 'task_b' tries to pull this value but gets None. What is the most likely cause?

A 'task_a' did not push any value to XCom.

B The key used in xcom_pull is misspelled.

C All of the above.

D 'task_b' is running before 'task_a' completes.

💡 Hint

Consider timing, key names, and whether the push actually happened.

If 'task_b' runs before 'task_a' finishes, the value won't be available yet. If the key is wrong or 'task_a' never pushed, the pull returns None. All these cause None.

🔀 Workflow

advanced

XCom Usage in Task Dependencies

Which Airflow DAG setup correctly ensures that a value pushed by 'task1' is available to 'task2' via XCom?

A 'task1' >> 'task2' with 'task2' pulling XCom using task_ids='task1'.

B 'task2' >> 'task1' with 'task2' pulling XCom using task_ids='task1'.

C 'task1' and 'task2' run in parallel with no dependencies.

D 'task1' pushes XCom but 'task2' does not pull it.

💡 Hint

Think about task order and how XCom pull uses task_ids.

Setting 'task1' upstream of 'task2' ensures 'task1' runs first and pushes the value. 'task2' can then pull it using task_ids='task1'.

✅ Best Practice

expert

Best Practice for Large Data in XCom

What is the best practice when you need to share large data between Airflow tasks using XCom?

A Store the large data in external storage (e.g., S3) and push only a reference or path via XCom.

B Push the entire large data directly via XCom for simplicity.

C Split the large data into many small pieces and push each piece separately via XCom.

D Avoid using XCom and rewrite tasks to not share data.

💡 Hint

Consider XCom size limits and performance.

XCom is not designed for large data. Best practice is to store big data externally and pass only a small reference or path through XCom to keep performance and reliability.

Practice

1. What does returning a value from a PythonOperator task do in Airflow?

easy

A. Automatically pushes the returned value to XCom for other tasks to use.

B. Saves the value only in the task logs without sharing.

C. Deletes any previous XCom values for that task.

D. Triggers the next task immediately without waiting.

Solution

1. Step 1: Understand PythonOperator return behavior

   When a PythonOperator returns a value, Airflow automatically stores it in XCom for that task instance.

2. Step 2: Recognize XCom purpose

   XCom is designed to share data between tasks, so returning a value pushes it to XCom for downstream tasks.

3. Final Answer:

   Automatically pushes the returned value to XCom for other tasks to use. -> Option A

4. Quick Check:

   Return value = XCom push [OK]

Hint: Return values from PythonOperator go to XCom automatically [OK]

Common Mistakes:

• Thinking return values are only logged, not shared
• Assuming return clears previous XCom data
• Believing return triggers next task immediately

2. Which of the following is the correct way to pull an XCom value in a downstream task?

easy

A. ti.xcom_pull(task_ids='task1', key='return_value')

B. ti.xcom_push(key='result')

C. xcom_pull(task_id='task1')

D. ti.pull_xcom(task_id='task1')

Solution

1. Step 1: Identify correct method signature

   The method to pull XCom data is ti.xcom_pull(), which requires task_ids and optionally a key.

2. Step 2: Match correct parameters

   Correct usage is ti.xcom_pull(task_ids='task1', key='return_value') to get the returned value from task1.

3. Final Answer:

   ti.xcom_pull(task_ids='task1', key='return_value') -> Option A

4. Quick Check:

   Use ti.xcom_pull with task_ids and key [OK]

Hint: Use ti.xcom_pull(task_ids, key) to get XCom data [OK]

Common Mistakes:

• Using xcom_push instead of xcom_pull to get data
• Wrong parameter names like task_id instead of task_ids
• Calling non-existent methods like ti.pull_xcom

3. Given this DAG snippet:

def task1():
    return {'value': 42}

def task2(ti):
    data = ti.xcom_pull(task_ids='task1')
    print(data)

What will task2 print when executed?

medium

A. None

B. {'return_value': {'value': 42}}

C. 42

D. Error: xcom_pull missing key

Solution

1. Step 1: Understand task1 return value

   task1 returns a dictionary {'value': 42}, which is pushed to XCom automatically.

2. Step 2: Analyze task2 xcom_pull call

   task2 calls ti.xcom_pull(task_ids='task1') without a key, so it pulls all XCom values from task1 as a dictionary {'return_value': {'value': 42}}.

3. Final Answer:

   {'return_value': {'value': 42}} -> Option B

4. Quick Check:

   Pull without key returns XComs dict [OK]

Hint: Pull without key returns XComs dict [OK]

Common Mistakes:

• Expecting only the inner value without specifying key
• Assuming None if key is not given
• Thinking it raises an error without key

4. What is wrong with this code snippet?

def task1():
    return 100

def task2(ti):
    value = ti.xcom_pull(task_ids='task1', key='value')
    print(value)

medium

A. task2 must use task_id instead of task_ids in xcom_pull.

B. task1 should push to XCom manually using xcom_push.

C. task2 pulls with wrong key; default return key is 'return_value'.

D. task1 return value must be a string, not int.

Solution

1. Step 1: Check task1 return and XCom key

   task1 returns 100, which is stored in XCom with the default key 'return_value'.

2. Step 2: Analyze task2 xcom_pull key usage

   task2 tries to pull with key='value', which does not exist, so it will return None.

3. Final Answer:

   task2 pulls with wrong key; default return key is 'return_value'. -> Option C

4. Quick Check:

   Default XCom key for return is 'return_value' [OK]

Hint: Default XCom key for return is 'return_value' [OK]

Common Mistakes:

• Thinking return requires manual xcom_push
• Using wrong key name in xcom_pull
• Confusing task_id and task_ids parameter names

5. You want to pass a dictionary from task1 to task3 through task2 without modifying task2. Which is the best way to do this using XCom return values?

hard

A. Use Airflow Variables instead of XCom for this data passing.

B. Have task1 push manually; task2 returns the same dictionary to pass it on.

C. Modify task2 to pull from task1 and push to task3 manually.

D. Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1').

Solution

1. Step 1: Understand XCom chaining

   XCom stores data per task instance; tasks can pull data from any previous task by specifying task_ids.

2. Step 2: Avoid modifying task2

   Since task2 should not be changed, task3 can pull data directly from task1's XCom return value.

3. Final Answer:

   Have task1 return the dictionary; task3 pulls it directly using ti.xcom_pull(task_ids='task1'). -> Option D

4. Quick Check:

   Pull directly from original task to avoid modifying intermediates [OK]

Hint: Pull XCom directly from original task to skip intermediate tasks [OK]

Common Mistakes:

• Thinking data must pass through every intermediate task
• Modifying tasks unnecessarily to forward XCom
• Using Variables instead of XCom for task-to-task data