XCom size limitations and alternatives in Apache Airflow - Step-by-Step Execution

Process Flow - XCom size limitations and alternatives

Task pushes XCom

↓

Check XCom size

↓

Store

↓

Task pulls XCom

↓

Process Data

This flow shows how Airflow tasks push data to XCom, check size limits, and either store it or use alternatives before pulling and processing.

Execution Sample

Apache Airflow

from airflow.models import XCom

def push_large_data(ti):
    data = 'x' * 1200000  # 1.2MB string
    ti.xcom_push(key='large_data', value=data)

This code tries to push a large string (1.2MB) to XCom, which exceeds the default size limit.

Process Table

Step	Action	Data Size (bytes)	Result	Notes
1	Task tries to push 1.2MB data to XCom	1200000	Fail	Exceeds default 48KB limit
2	Fallback: Store data in external storage (e.g., S3)	1200000	Success	Data stored externally
3	Push XCom with pointer (e.g., S3 path)	100	Success	Small pointer stored in XCom
4	Task pulls XCom pointer	100	Success	Pointer retrieved
5	Task fetches large data from external storage	1200000	Success	Data retrieved for processing
6	Process data	1200000	Success	Data processed
7	End	-	-	Execution complete

💡 XCom push fails due to size limit; alternative external storage used with pointer in XCom

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
data	''	'x'*1200000	'x'*1200000 stored externally	S3 path string	S3 path string	'x'*1200000 fetched	'x'*1200000 processed
XCom_value	None	Fail to store large data	None	S3 path string stored	S3 path string retrieved	S3 path string	S3 path string

Key Moments - 3 Insights

Why does pushing large data directly to XCom fail?

Because XCom has a default size limit (~48KB). Step 1 in the execution_table shows the push fails due to exceeding this limit.

How can large data be passed between tasks if XCom size is limited?

By storing the large data externally (like in S3) and pushing only a small pointer or path in XCom, as shown in steps 2 and 3.

What does the task do to retrieve the large data after getting the pointer from XCom?

It uses the pointer from XCom to fetch the large data from external storage before processing, as shown in steps 4 and 5.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the XCom push fail due to size?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Result' column for failure in step 1.

According to variable_tracker, what is stored in XCom after step 3?

A The full 1.2MB data string

B An empty string

C A small pointer string (e.g., S3 path)

D None

💡 Hint

Look at the 'XCom_value' row after step 3 in variable_tracker.

If the data size was under 48KB, how would the execution_table change?

A Step 1 would fail and steps 2 and 3 would run

B Step 1 would succeed and steps 2 and 3 would be skipped

C Step 5 would fail

D Step 4 would fail

💡 Hint

Refer to the exit_note and steps showing failure due to size.

Concept Snapshot

XCom stores small data between Airflow tasks.
Default size limit is ~48KB.
Large data push fails and needs alternatives.
Use external storage (S3, DB) for big data.
Push a small pointer in XCom instead.
Tasks pull pointer, then fetch large data externally.

Full Transcript

In Airflow, tasks communicate using XCom, which has a size limit of about 48KB. If a task tries to push data larger than this limit, the push fails. To handle large data, the task can store the data externally, such as in S3, and push only a small pointer or path to that data in XCom. The downstream task then pulls this pointer from XCom and fetches the large data from the external storage before processing it. This approach avoids XCom size limitations and ensures smooth data passing between tasks.

Practice

1. What is the main reason to avoid storing large data directly in Airflow's XCom?

easy

A. Because XCom automatically deletes large data after one hour

B. Because Airflow does not support any data sharing between tasks

C. Because large data can slow down Airflow and cause stability issues

D. Because large data is encrypted and cannot be read by tasks

Solution

1. Step 1: Understand XCom's purpose

   XCom is designed for small data sharing between tasks, not large data storage.

2. Step 2: Recognize impact of large data in XCom

   Storing large data in XCom can slow down Airflow and cause stability problems.

3. Final Answer:

   Because large data can slow down Airflow and cause stability issues -> Option C

4. Quick Check:

   XCom size limits = Avoid large data [OK]

Hint: Remember: XCom is for small data only [OK]

Common Mistakes:

• Thinking XCom can handle large files safely
• Believing XCom auto-deletes large data quickly
• Assuming Airflow does not support task communication

2. Which of the following is the correct way to push a small value to XCom in an Airflow PythonOperator task?

easy

A. task_instance.push_xcom('result', 123)

B. task_instance.xcom_push(key='result', value=123)

C. xcom_push(task_instance, 'result', 123)

D. task_instance.xcom('result', 123)

Solution

1. Step 1: Recall XCom push method syntax

   The correct method to push data is task_instance.xcom_push(key, value).

2. Step 2: Match the correct syntax

   task_instance.xcom_push(key='result', value=123) uses the correct method and parameter names.

3. Final Answer:

   task_instance.xcom_push(key='result', value=123) -> Option B

4. Quick Check:

   XCom push method = xcom_push(key, value) [OK]

Hint: Use xcom_push with key and value parameters [OK]

Common Mistakes:

• Using incorrect method names like push_xcom
• Omitting key or value parameters
• Calling xcom as a method instead of xcom_push

3. Given the following code snippet in an Airflow task, what will be the output when retrieving the XCom value?

def push_func(ti):
    ti.xcom_push(key='data', value='small string')

def pull_func(ti):
    return ti.xcom_pull(key='data', task_ids='push_task')

# Assume push_func runs first and then pull_func runs
result = pull_func(ti)

medium

A. Empty string

B. None

C. KeyError

D. 'small string'

Solution

1. Step 1: Understand XCom push and pull sequence

   The first task pushes 'small string' with key 'data'. The second task pulls it using the same key and task id.

2. Step 2: Predict the pull result

   Since the key and task id match, the pull returns the pushed value 'small string'.

3. Final Answer:

   'small string' -> Option D

4. Quick Check:

   XCom pull returns pushed value if keys match [OK]

Hint: Pull returns pushed value if keys and task_ids match [OK]

Common Mistakes:

• Expecting None if tasks run in order
• Confusing key names or task_ids
• Assuming pull returns error if no data

4. You tried to push a large file content directly into XCom but Airflow throws an error. What is the best way to fix this issue?

medium

A. Store the large file in external storage and push only its reference in XCom

B. Increase Airflow's XCom size limit in the configuration file

C. Split the large file into many small pieces and push each piece separately in XCom

D. Disable XCom size checks in Airflow's source code

Solution

1. Step 1: Identify the cause of the error

   Airflow limits XCom size; pushing large files directly causes errors.

2. Step 2: Choose the best practice alternative

   Storing large data externally and pushing only a reference keeps Airflow stable and avoids errors.

3. Final Answer:

   Store the large file in external storage and push only its reference in XCom -> Option A

4. Quick Check:

   Large data in XCom? Use external storage reference [OK]

Hint: Use external storage for large data, push reference only [OK]

Common Mistakes:

• Trying to increase XCom size limit (not supported)
• Splitting data manually is complex and error-prone
• Modifying Airflow source code is risky and unnecessary

5. You have a DAG where a task generates a large JSON report. You want to pass this report to the next task without slowing down Airflow. Which approach is best?

hard

A. Save the JSON report to an S3 bucket and push the S3 path via XCom

B. Serialize the JSON report and push it directly in XCom

C. Convert the JSON to a string and push it in XCom without size check

D. Store the JSON report in Airflow variables and read it in the next task

Solution

1. Step 1: Recognize XCom size limits

   Large JSON data should not be pushed directly to XCom to avoid performance issues.

2. Step 2: Select best alternative for large data sharing

   Saving the report externally (e.g., S3) and pushing only the path keeps Airflow fast and stable.

3. Final Answer:

   Save the JSON report to an S3 bucket and push the S3 path via XCom -> Option A

4. Quick Check:

   Large data? Use external storage + XCom reference [OK]

Hint: Push large data externally, pass only reference in XCom [OK]

Common Mistakes:

• Pushing large JSON directly in XCom
• Ignoring Airflow variables size limits
• Assuming string conversion bypasses size limits