XCom size limitations and alternatives in Apache Airflow - Mini Project: Build & Apply

XCom Size Limitations and Alternatives in Airflow

📖 Scenario: You are working with Apache Airflow to automate data workflows. You want to pass data between tasks using XComs, but you learn that XComs have size limits and storing large data directly can cause issues.This project will guide you through creating a simple Airflow DAG that demonstrates the size limitation of XComs and shows an alternative approach by storing large data in a file and passing only the file path via XCom.

🎯 Goal: Build an Airflow DAG with two tasks:A task that tries to push a large data object to XCom (which is not recommended).A task that pushes a file path to XCom instead, demonstrating a better alternative for large data.You will learn how to handle XCom size limits and use alternatives effectively.

📋 What You'll Learn

Create a Python dictionary with a large data string

Create a variable for the file path to store large data

Push large data directly to XCom in one task

Push file path to XCom in another task

Print the XCom values in the final step

💡 Why This Matters

🌍 Real World

In real Airflow workflows, passing large data directly via XCom can cause failures or slowdowns. Using file paths or external storage is a common practice.

💼 Career

Understanding XCom size limits and alternatives is important for building reliable and scalable data pipelines in Airflow, a key skill for DevOps and data engineering roles.

1

Create a large data dictionary

Create a Python dictionary called large_data with a key 'data' and a value that is a string of 10000 'A' characters.

Apache Airflow

# Create the large_data dictionary with 10000 'A' characters
# Your code here

Hint

Use string multiplication like 'A' * 10000 to create a long string.

2

Create a file path variable for large data

Create a variable called file_path and set it to the string '/tmp/large_data.txt'.

Apache Airflow

large_data = {'data': 'A' * 10000}
# Create the file_path variable with '/tmp/large_data.txt'
# Your code here

Hint

Just assign the string path to the variable file_path.

3

Push large data and file path to XCom in tasks

Write two Python functions: push_large_data and push_file_path. In push_large_data, push large_data to XCom using ti.xcom_push(key='large_data', value=large_data). In push_file_path, push file_path to XCom using ti.xcom_push(key='file_path', value=file_path).

Apache Airflow

large_data = {'data': 'A' * 10000}
file_path = '/tmp/large_data.txt'

# Define push_large_data function
# Your code here

# Define push_file_path function
# Your code here

Hint

Use the ti parameter to push data to XCom with ti.xcom_push.

4

Print XCom values in a task

Write a function called print_xcom_values that pulls the large_data and file_path from XCom using ti.xcom_pull(key='large_data') and ti.xcom_pull(key='file_path'). Then print both values exactly as shown: print('Large data length:', len(large_data['data'])) and print('File path:', file_path).

Apache Airflow

large_data = {'data': 'A' * 10000}
file_path = '/tmp/large_data.txt'

def push_large_data(ti):
    ti.xcom_push(key='large_data', value=large_data)

def push_file_path(ti):
    ti.xcom_push(key='file_path', value=file_path)

# Define print_xcom_values function
# Your code here

Hint

Use ti.xcom_pull to get values and print to show them.

Practice

1. What is the main reason to avoid storing large data directly in Airflow's XCom?

easy

A. Because XCom automatically deletes large data after one hour

B. Because Airflow does not support any data sharing between tasks

C. Because large data can slow down Airflow and cause stability issues

D. Because large data is encrypted and cannot be read by tasks

Solution

1. Step 1: Understand XCom's purpose

   XCom is designed for small data sharing between tasks, not large data storage.

2. Step 2: Recognize impact of large data in XCom

   Storing large data in XCom can slow down Airflow and cause stability problems.

3. Final Answer:

   Because large data can slow down Airflow and cause stability issues -> Option C

4. Quick Check:

   XCom size limits = Avoid large data [OK]

Hint: Remember: XCom is for small data only [OK]

Common Mistakes:

• Thinking XCom can handle large files safely
• Believing XCom auto-deletes large data quickly
• Assuming Airflow does not support task communication

2. Which of the following is the correct way to push a small value to XCom in an Airflow PythonOperator task?

easy

A. task_instance.push_xcom('result', 123)

B. task_instance.xcom_push(key='result', value=123)

C. xcom_push(task_instance, 'result', 123)

D. task_instance.xcom('result', 123)

Solution

1. Step 1: Recall XCom push method syntax

   The correct method to push data is task_instance.xcom_push(key, value).

2. Step 2: Match the correct syntax

   task_instance.xcom_push(key='result', value=123) uses the correct method and parameter names.

3. Final Answer:

   task_instance.xcom_push(key='result', value=123) -> Option B

4. Quick Check:

   XCom push method = xcom_push(key, value) [OK]

Hint: Use xcom_push with key and value parameters [OK]

Common Mistakes:

• Using incorrect method names like push_xcom
• Omitting key or value parameters
• Calling xcom as a method instead of xcom_push

3. Given the following code snippet in an Airflow task, what will be the output when retrieving the XCom value?

def push_func(ti):
    ti.xcom_push(key='data', value='small string')

def pull_func(ti):
    return ti.xcom_pull(key='data', task_ids='push_task')

# Assume push_func runs first and then pull_func runs
result = pull_func(ti)

medium

A. Empty string

B. None

C. KeyError

D. 'small string'

Solution

1. Step 1: Understand XCom push and pull sequence

   The first task pushes 'small string' with key 'data'. The second task pulls it using the same key and task id.

2. Step 2: Predict the pull result

   Since the key and task id match, the pull returns the pushed value 'small string'.

3. Final Answer:

   'small string' -> Option D

4. Quick Check:

   XCom pull returns pushed value if keys match [OK]

Hint: Pull returns pushed value if keys and task_ids match [OK]

Common Mistakes:

• Expecting None if tasks run in order
• Confusing key names or task_ids
• Assuming pull returns error if no data

4. You tried to push a large file content directly into XCom but Airflow throws an error. What is the best way to fix this issue?

medium

A. Store the large file in external storage and push only its reference in XCom

B. Increase Airflow's XCom size limit in the configuration file

C. Split the large file into many small pieces and push each piece separately in XCom

D. Disable XCom size checks in Airflow's source code

Solution

1. Step 1: Identify the cause of the error

   Airflow limits XCom size; pushing large files directly causes errors.

2. Step 2: Choose the best practice alternative

   Storing large data externally and pushing only a reference keeps Airflow stable and avoids errors.

3. Final Answer:

   Store the large file in external storage and push only its reference in XCom -> Option A

4. Quick Check:

   Large data in XCom? Use external storage reference [OK]

Hint: Use external storage for large data, push reference only [OK]

Common Mistakes:

• Trying to increase XCom size limit (not supported)
• Splitting data manually is complex and error-prone
• Modifying Airflow source code is risky and unnecessary

5. You have a DAG where a task generates a large JSON report. You want to pass this report to the next task without slowing down Airflow. Which approach is best?

hard

A. Save the JSON report to an S3 bucket and push the S3 path via XCom

B. Serialize the JSON report and push it directly in XCom

C. Convert the JSON to a string and push it in XCom without size check

D. Store the JSON report in Airflow variables and read it in the next task

Solution

1. Step 1: Recognize XCom size limits

   Large JSON data should not be pushed directly to XCom to avoid performance issues.

2. Step 2: Select best alternative for large data sharing

   Saving the report externally (e.g., S3) and pushing only the path keeps Airflow fast and stable.

3. Final Answer:

   Save the JSON report to an S3 bucket and push the S3 path via XCom -> Option A

4. Quick Check:

   Large data? Use external storage + XCom reference [OK]

Hint: Push large data externally, pass only reference in XCom [OK]

Common Mistakes:

• Pushing large JSON directly in XCom
• Ignoring Airflow variables size limits
• Assuming string conversion bypasses size limits