Pushing and pulling XCom values in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

What is the output of pulling an XCom value?

Given a task pushes an XCom value with key 'result' and value 42, what will be the output of pulling this XCom in another task?

Apache Airflow

value = ti.xcom_pull(task_ids='push_task', key='result')
print(value)

A None

B '' (empty string)

C KeyError

D 42

💡 Hint

Remember that xcom_pull returns the value pushed with the matching key and task_id.

The ti.xcom_pull method retrieves the value pushed by the specified task and key. Since the value 42 was pushed with key 'result', pulling it returns 42.

🧠 Conceptual

intermediate

Which method is used to push an XCom value in Airflow?

You want to share data from one task to another using XComs. Which method should you use inside a task to push a value?

A ti.xcom_push(key='my_key', value='data')

B push_xcom(ti, key='my_key', value='data')

C xcom_push(key='my_key', value='data')

D ti.push_xcom(key='my_key', value='data')

💡 Hint

The method is called on the task instance object (ti).

The correct method to push an XCom value is ti.xcom_push(). It takes a key and a value to store.

❓ Troubleshoot

advanced

Why does xcom_pull return None unexpectedly?

You have two tasks: 'push_task' pushes an XCom with key 'data'. 'pull_task' tries to pull it but gets None. What is a likely cause?

A 'pull_task' runs before 'push_task' completes

B The key used in xcom_pull is misspelled

C All of the above

D 'push_task' did not call ti.xcom_push

💡 Hint

Consider timing, spelling, and whether the push actually happened.

All these reasons can cause ti.xcom_pull to return None: pulling before push, wrong key, or no push call.

🔀 Workflow

advanced

Order of operations for pushing and pulling XComs

Arrange the steps in the correct order to share data between two Airflow tasks using XComs.

A 1,3,4,2

B 3,1,4,2

C 1,4,3,2

D 3,4,1,2

💡 Hint

Pushing happens during Task A execution, which must complete before Task B runs and pulls.

Task A pushes the value (1), then completes (3). Task B runs after Task A (4) and pulls the value (2).

✅ Best Practice

expert

What is the recommended way to avoid XCom data size issues?

XComs have size limits. Which approach is best to handle large data sharing between tasks?

A Compress data and push compressed bytes in XCom

B Store large data in external storage and push only a reference via XCom

C Split large data into many small XCom pushes

D Increase Airflow database size limits to accommodate large XComs

💡 Hint

Think about scalability and Airflow best practices.

Best practice is to store large data outside Airflow (e.g., S3, database) and push only a pointer or filename via XCom to avoid database bloat and performance issues.

Practice

1. What is the main purpose of using xcom_push and xcom_pull in Airflow?

easy

A. To share small pieces of data between tasks in a workflow

B. To schedule tasks to run at specific times

C. To create new DAGs dynamically

D. To monitor task execution logs

Solution

1. Step 1: Understand the role of XComs

   XComs are designed to let tasks exchange small data pieces during workflow execution.

2. Step 2: Identify the functions' purpose

   xcom_push sends data, and xcom_pull retrieves it between tasks.

3. Final Answer:

   To share small pieces of data between tasks in a workflow -> Option A

4. Quick Check:

   Data sharing = A [OK]

Hint: Remember: push sends data, pull gets data [OK]

Common Mistakes:

• Confusing XComs with scheduling tasks
• Thinking XComs create DAGs
• Assuming XComs handle logs

2. Which of the following is the correct syntax to push a value '42' with key 'result' inside a task instance ti?

easy

A. xcom_push(ti, 'result', 42)

B. ti.push_xcom('result', 42)

C. ti.xcom_pull(key='result', value=42)

D. ti.xcom_push(key='result', value=42)

Solution

1. Step 1: Recall the method signature

   The correct method to push data is ti.xcom_push(key=..., value=...).

2. Step 2: Check each option

   ti.xcom_push(key='result', value=42) matches the correct syntax. Options A and B use wrong method names or argument order. ti.xcom_pull(key='result', value=42) uses xcom_pull which is for pulling, not pushing.

3. Final Answer:

   ti.xcom_push(key='result', value=42) -> Option D

4. Quick Check:

   Push syntax = C [OK]

Hint: Push uses ti.xcom_push(key=..., value=...) [OK]

Common Mistakes:

• Using wrong method names like push_xcom
• Confusing push with pull
• Incorrect argument order

3. Given the following code snippet inside a task:

ti.xcom_push(key='data', value=100)
value = ti.xcom_pull(key='data')

What will be the value of the variable value after execution?

medium

A. None

B. KeyError

C. 100

D. 'data'

Solution

1. Step 1: Understand the push operation

   The code pushes the integer 100 with key 'data' into XCom storage.

2. Step 2: Understand the pull operation

   The code pulls the value stored with key 'data', which is 100.

3. Final Answer:

   100 -> Option C

4. Quick Check:

   Push 100, pull 100 = D [OK]

Hint: Pull returns the value pushed with the same key [OK]

Common Mistakes:

• Expecting pull to return None if key exists
• Confusing key with value
• Assuming pull raises error if key exists

4. You wrote this code inside a task:

ti.xcom_push('result', 10)
value = ti.xcom_pull(key='result')

But value is always None. What is the likely problem?

medium

A. The xcom_push call is missing keyword arguments

B. The key 'result' does not exist in XCom

C. You must call xcom_pull before xcom_push

D. XComs cannot store integers

Solution

1. Step 1: Check the xcom_push syntax

   The method requires keyword arguments: key= and value=. Passing positional arguments causes failure.

2. Step 2: Understand the effect of wrong syntax

   Because the push failed silently, the key 'result' was never stored, so pull returns None.

3. Final Answer:

   The xcom_push call is missing keyword arguments -> Option A

4. Quick Check:

   Push needs key= and value= [OK]

Hint: Always use key= and value= in xcom_push [OK]

Common Mistakes:

• Using positional arguments in xcom_push
• Assuming pull before push order matters
• Thinking XComs can't store integers

5. You want to pass a dictionary from one task to another using XCom. Which code snippet correctly pushes and pulls the dictionary {'a': 1, 'b': 2}?

hard

A. ti.xcom_push('my_dict', {'a': 1, 'b': 2})
data = ti.xcom_pull('my_dict')

B. ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2})
data = ti.xcom_pull(key='my_dict')

C. ti.xcom_push(value={'a': 1, 'b': 2}, key='my_dict')
data = ti.xcom_pull(key='my_dict')

D. ti.xcom_push(key='my_dict', value='{'a': 1, 'b': 2}')
data = ti.xcom_pull(key='my_dict')

Solution

1. Step 1: Verify correct xcom_push syntax with dictionary

   ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2}) data = ti.xcom_pull(key='my_dict') uses correct keyword arguments key= and value= with a dictionary value.

2. Step 2: Check other options for errors

   ti.xcom_push(value={'a': 1, 'b': 2}, key='my_dict') data = ti.xcom_pull(key='my_dict') uses non-standard keyword argument order; ti.xcom_push('my_dict', {'a': 1, 'b': 2}) data = ti.xcom_pull('my_dict') uses positional args which is incorrect; ti.xcom_push(key='my_dict', value='{'a': 1, 'b': 2}') data = ti.xcom_pull(key='my_dict') passes dictionary as a string with wrong quotes causing syntax error.

3. Final Answer:

   ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2}) data = ti.xcom_pull(key='my_dict') -> Option B

4. Quick Check:

   Push dict with key= and value= [OK]

Hint: Use key= and value= with dict, avoid string quotes errors [OK]

Common Mistakes:

• Using positional arguments in xcom_push
• Passing dict as a string with wrong quotes
• Mixing argument order incorrectly