XCom size limitations and alternatives in Apache Airflow - Step-by-Step Execution

Process Flow - XCom size limitations and alternatives

Task pushes XCom

↓

Check XCom size

↓

Store

↓

Task pulls XCom

↓

Process Data

This flow shows how Airflow tasks push data to XCom, check size limits, and either store it or use alternatives before pulling and processing.

Execution Sample

Apache Airflow

from airflow.models import XCom

def push_large_data(ti):
    data = 'x' * 1200000  # 1.2MB string
    ti.xcom_push(key='large_data', value=data)

This code tries to push a large string (1.2MB) to XCom, which exceeds the default size limit.

Process Table

Step	Action	Data Size (bytes)	Result	Notes
1	Task tries to push 1.2MB data to XCom	1200000	Fail	Exceeds default 48KB limit
2	Fallback: Store data in external storage (e.g., S3)	1200000	Success	Data stored externally
3	Push XCom with pointer (e.g., S3 path)	100	Success	Small pointer stored in XCom
4	Task pulls XCom pointer	100	Success	Pointer retrieved
5	Task fetches large data from external storage	1200000	Success	Data retrieved for processing
6	Process data	1200000	Success	Data processed
7	End	-	-	Execution complete

💡 XCom push fails due to size limit; alternative external storage used with pointer in XCom

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
data	''	'x'*1200000	'x'*1200000 stored externally	S3 path string	S3 path string	'x'*1200000 fetched	'x'*1200000 processed
XCom_value	None	Fail to store large data	None	S3 path string stored	S3 path string retrieved	S3 path string	S3 path string

Key Moments - 3 Insights

Why does pushing large data directly to XCom fail?

Because XCom has a default size limit (~48KB). Step 1 in the execution_table shows the push fails due to exceeding this limit.

How can large data be passed between tasks if XCom size is limited?

By storing the large data externally (like in S3) and pushing only a small pointer or path in XCom, as shown in steps 2 and 3.

What does the task do to retrieve the large data after getting the pointer from XCom?

It uses the pointer from XCom to fetch the large data from external storage before processing, as shown in steps 4 and 5.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the XCom push fail due to size?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Check the 'Result' column for failure in step 1.

According to variable_tracker, what is stored in XCom after step 3?

A The full 1.2MB data string

B An empty string

C A small pointer string (e.g., S3 path)

D None

💡 Hint

Look at the 'XCom_value' row after step 3 in variable_tracker.

If the data size was under 48KB, how would the execution_table change?

A Step 1 would fail and steps 2 and 3 would run

B Step 1 would succeed and steps 2 and 3 would be skipped

C Step 5 would fail

D Step 4 would fail

💡 Hint

Refer to the exit_note and steps showing failure due to size.

Concept Snapshot

XCom stores small data between Airflow tasks.
Default size limit is ~48KB.
Large data push fails and needs alternatives.
Use external storage (S3, DB) for big data.
Push a small pointer in XCom instead.
Tasks pull pointer, then fetch large data externally.

Full Transcript

In Airflow, tasks communicate using XCom, which has a size limit of about 48KB. If a task tries to push data larger than this limit, the push fails. To handle large data, the task can store the data externally, such as in S3, and push only a small pointer or path to that data in XCom. The downstream task then pulls this pointer from XCom and fetches the large data from the external storage before processing it. This approach avoids XCom size limitations and ensures smooth data passing between tasks.