Pushing and pulling XCom values in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is an XCom in Airflow?

An XCom (short for cross-communication) is a way for tasks in Airflow to share small pieces of data or messages between each other during a workflow run.

beginner

How do you push an XCom value in a PythonOperator task?

You can push an XCom by returning a value from the Python function used in the PythonOperator. Airflow automatically pushes the return value as an XCom.

intermediate

How can you pull an XCom value from another task?

Use the task instance's xcom_pull method with the task_id of the task that pushed the value. For example: ti.xcom_pull(task_ids='task_id').

intermediate

What parameters can you use with xcom_pull to get specific data?

You can specify task_ids, key (default is 'return_value'), and include_prior_dates to control which XCom value you want to pull.

advanced

Can you push an XCom manually without returning a value?

Yes, by using the xcom_push method inside a task's context, you can push any key-value pair as an XCom without returning it.

What does Airflow automatically do when a PythonOperator's function returns a value?

A Deletes the return value

B Pushes the return value as an XCom

C Saves the return value to a file

D Ignores the return value

Airflow automatically pushes the return value of a PythonOperator's function as an XCom for other tasks to use.

Which method is used to retrieve an XCom value from another task?

A xcom_pull

B xcom_push

C get_xcom

D fetch_xcom

The xcom_pull method is used to get XCom values from other tasks.

What is the default key used when pulling an XCom value if no key is specified?

A return_value

B default

C result

D value

The default key for XCom values is 'return_value' when no key is specified.

How can you push an XCom value manually inside a task?

A Using print statement

B Returning the value from the function

C Using ti.xcom_push(key, value)

D Using ti.xcom_pull()

You can manually push an XCom using ti.xcom_push with a key and value.

Which of the following is NOT a valid parameter for xcom_pull?

A task_ids

B key

C include_prior_dates

D push_value

'push_value' is not a parameter for xcom_pull; the others are valid.

Explain how to share data between two tasks in Airflow using XComs.

Think about how one task sends and another receives small data pieces.

Describe the difference between pushing XComs by returning a value and manually pushing with xcom_push.

Consider when you want to send multiple or named pieces of data.

Practice

1. What is the main purpose of using xcom_push and xcom_pull in Airflow?

easy

A. To share small pieces of data between tasks in a workflow

B. To schedule tasks to run at specific times

C. To create new DAGs dynamically

D. To monitor task execution logs

Solution

1. Step 1: Understand the role of XComs

   XComs are designed to let tasks exchange small data pieces during workflow execution.

2. Step 2: Identify the functions' purpose

   xcom_push sends data, and xcom_pull retrieves it between tasks.

3. Final Answer:

   To share small pieces of data between tasks in a workflow -> Option A

4. Quick Check:

   Data sharing = A [OK]

Hint: Remember: push sends data, pull gets data [OK]

Common Mistakes:

• Confusing XComs with scheduling tasks
• Thinking XComs create DAGs
• Assuming XComs handle logs

2. Which of the following is the correct syntax to push a value '42' with key 'result' inside a task instance ti?

easy

A. xcom_push(ti, 'result', 42)

B. ti.push_xcom('result', 42)

C. ti.xcom_pull(key='result', value=42)

D. ti.xcom_push(key='result', value=42)

Solution

1. Step 1: Recall the method signature

   The correct method to push data is ti.xcom_push(key=..., value=...).

2. Step 2: Check each option

   ti.xcom_push(key='result', value=42) matches the correct syntax. Options A and B use wrong method names or argument order. ti.xcom_pull(key='result', value=42) uses xcom_pull which is for pulling, not pushing.

3. Final Answer:

   ti.xcom_push(key='result', value=42) -> Option D

4. Quick Check:

   Push syntax = C [OK]

Hint: Push uses ti.xcom_push(key=..., value=...) [OK]

Common Mistakes:

• Using wrong method names like push_xcom
• Confusing push with pull
• Incorrect argument order

3. Given the following code snippet inside a task:

ti.xcom_push(key='data', value=100)
value = ti.xcom_pull(key='data')

What will be the value of the variable value after execution?

medium

A. None

B. KeyError

C. 100

D. 'data'

Solution

1. Step 1: Understand the push operation

   The code pushes the integer 100 with key 'data' into XCom storage.

2. Step 2: Understand the pull operation

   The code pulls the value stored with key 'data', which is 100.

3. Final Answer:

   100 -> Option C

4. Quick Check:

   Push 100, pull 100 = D [OK]

Hint: Pull returns the value pushed with the same key [OK]

Common Mistakes:

• Expecting pull to return None if key exists
• Confusing key with value
• Assuming pull raises error if key exists

4. You wrote this code inside a task:

ti.xcom_push('result', 10)
value = ti.xcom_pull(key='result')

But value is always None. What is the likely problem?

medium

A. The xcom_push call is missing keyword arguments

B. The key 'result' does not exist in XCom

C. You must call xcom_pull before xcom_push

D. XComs cannot store integers

Solution

1. Step 1: Check the xcom_push syntax

   The method requires keyword arguments: key= and value=. Passing positional arguments causes failure.

2. Step 2: Understand the effect of wrong syntax

   Because the push failed silently, the key 'result' was never stored, so pull returns None.

3. Final Answer:

   The xcom_push call is missing keyword arguments -> Option A

4. Quick Check:

   Push needs key= and value= [OK]

Hint: Always use key= and value= in xcom_push [OK]

Common Mistakes:

• Using positional arguments in xcom_push
• Assuming pull before push order matters
• Thinking XComs can't store integers

5. You want to pass a dictionary from one task to another using XCom. Which code snippet correctly pushes and pulls the dictionary {'a': 1, 'b': 2}?

hard

A. ti.xcom_push('my_dict', {'a': 1, 'b': 2})
data = ti.xcom_pull('my_dict')

B. ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2})
data = ti.xcom_pull(key='my_dict')

C. ti.xcom_push(value={'a': 1, 'b': 2}, key='my_dict')
data = ti.xcom_pull(key='my_dict')

D. ti.xcom_push(key='my_dict', value='{'a': 1, 'b': 2}')
data = ti.xcom_pull(key='my_dict')

Solution

1. Step 1: Verify correct xcom_push syntax with dictionary

   ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2}) data = ti.xcom_pull(key='my_dict') uses correct keyword arguments key= and value= with a dictionary value.

2. Step 2: Check other options for errors

   ti.xcom_push(value={'a': 1, 'b': 2}, key='my_dict') data = ti.xcom_pull(key='my_dict') uses non-standard keyword argument order; ti.xcom_push('my_dict', {'a': 1, 'b': 2}) data = ti.xcom_pull('my_dict') uses positional args which is incorrect; ti.xcom_push(key='my_dict', value='{'a': 1, 'b': 2}') data = ti.xcom_pull(key='my_dict') passes dictionary as a string with wrong quotes causing syntax error.

3. Final Answer:

   ti.xcom_push(key='my_dict', value={'a': 1, 'b': 2}) data = ti.xcom_pull(key='my_dict') -> Option B

4. Quick Check:

   Push dict with key= and value= [OK]

Hint: Use key= and value= with dict, avoid string quotes errors [OK]

Common Mistakes:

• Using positional arguments in xcom_push
• Passing dict as a string with wrong quotes
• Mixing argument order incorrectly