XCom size limitations and alternatives in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why does Airflow limit XCom size?

Airflow has a default size limit for XCom data. Why is this limit important?

A To stop tasks from running longer than their timeout

B To prevent the metadata database from growing too large and slowing down the scheduler

C To avoid tasks from failing due to memory overflow in worker nodes

D To ensure XCom data is encrypted automatically

💡 Hint

Think about what happens if too much data is stored in the Airflow metadata database.

Airflow stores XCom data in its metadata database. Large XComs can cause the database to grow quickly, which slows down the scheduler and affects performance.

💻 Command Output

intermediate

What happens when an XCom exceeds the size limit?

Given a task that tries to push an XCom value larger than the default limit, what is the expected behavior?

Apache Airflow

task_instance.xcom_push(key='large_data', value='x' * 5000000)  # 5MB string

A The scheduler crashes due to database overload

B The XCom is silently truncated to the allowed size and task succeeds

C The task succeeds and the full data is stored without error

D The task fails with an AirflowException about XCom size limit

💡 Hint

Consider what Airflow does to protect the metadata database from large data.

When an XCom value exceeds the size limit, Airflow raises an AirflowException to prevent storing too large data.

✅ Best Practice

advanced

Best alternative to large XCom data

If you need to share large data between tasks in Airflow, which approach is best to avoid XCom size limits?

A Store the data in an external storage like S3 or a database and pass the reference via XCom

B Split the data into many small XComs under the size limit

C Compress the data and push it as a single XCom

D Increase the Airflow XCom size limit in the configuration

💡 Hint

Think about scalable and reliable ways to share large data between tasks.

Storing large data externally and passing only a reference in XCom keeps the metadata database small and avoids size limits.

❓ Troubleshoot

advanced

Diagnosing XCom size limit errors

You see a task failing with an error mentioning XCom size limit. What is the most effective first step to fix this?

A Modify the task to store large data externally and push only a reference in XCom

B Restart the Airflow scheduler to clear the error

C Increase the worker memory to handle larger XComs

D Delete old XCom entries from the metadata database manually

💡 Hint

Think about the root cause of the error and how to avoid it.

The error occurs because the task tries to push too large data. Changing the task to store data externally and push a reference fixes the root cause.

🔀 Workflow

expert

Design a workflow to handle large data passing between tasks

You have a DAG where Task A generates a 100MB file and Task B needs to process it. How should you design the data passing to avoid XCom size limits?

A Task A encodes the file content as base64 and pushes it via XCom; Task B decodes it

B Task A splits the file into 10MB chunks and pushes each chunk as separate XComs; Task B reassembles them

C Task A uploads the file to S3 and pushes the S3 path via XCom; Task B downloads from S3 using the path

D Task A pushes the file content directly via XCom; Task B reads it from XCom

💡 Hint

Consider scalability, reliability, and Airflow best practices for large data.

Uploading large files to external storage and passing only the path via XCom is the recommended approach to handle large data efficiently.

Practice

1. What is the main reason to avoid storing large data directly in Airflow's XCom?

easy

A. Because XCom automatically deletes large data after one hour

B. Because Airflow does not support any data sharing between tasks

C. Because large data can slow down Airflow and cause stability issues

D. Because large data is encrypted and cannot be read by tasks

Solution

1. Step 1: Understand XCom's purpose

   XCom is designed for small data sharing between tasks, not large data storage.

2. Step 2: Recognize impact of large data in XCom

   Storing large data in XCom can slow down Airflow and cause stability problems.

3. Final Answer:

   Because large data can slow down Airflow and cause stability issues -> Option C

4. Quick Check:

   XCom size limits = Avoid large data [OK]

Hint: Remember: XCom is for small data only [OK]

Common Mistakes:

• Thinking XCom can handle large files safely
• Believing XCom auto-deletes large data quickly
• Assuming Airflow does not support task communication

2. Which of the following is the correct way to push a small value to XCom in an Airflow PythonOperator task?

easy

A. task_instance.push_xcom('result', 123)

B. task_instance.xcom_push(key='result', value=123)

C. xcom_push(task_instance, 'result', 123)

D. task_instance.xcom('result', 123)

Solution

1. Step 1: Recall XCom push method syntax

   The correct method to push data is task_instance.xcom_push(key, value).

2. Step 2: Match the correct syntax

   task_instance.xcom_push(key='result', value=123) uses the correct method and parameter names.

3. Final Answer:

   task_instance.xcom_push(key='result', value=123) -> Option B

4. Quick Check:

   XCom push method = xcom_push(key, value) [OK]

Hint: Use xcom_push with key and value parameters [OK]

Common Mistakes:

• Using incorrect method names like push_xcom
• Omitting key or value parameters
• Calling xcom as a method instead of xcom_push

3. Given the following code snippet in an Airflow task, what will be the output when retrieving the XCom value?

def push_func(ti):
    ti.xcom_push(key='data', value='small string')

def pull_func(ti):
    return ti.xcom_pull(key='data', task_ids='push_task')

# Assume push_func runs first and then pull_func runs
result = pull_func(ti)

medium

A. Empty string

B. None

C. KeyError

D. 'small string'

Solution

1. Step 1: Understand XCom push and pull sequence

   The first task pushes 'small string' with key 'data'. The second task pulls it using the same key and task id.

2. Step 2: Predict the pull result

   Since the key and task id match, the pull returns the pushed value 'small string'.

3. Final Answer:

   'small string' -> Option D

4. Quick Check:

   XCom pull returns pushed value if keys match [OK]

Hint: Pull returns pushed value if keys and task_ids match [OK]

Common Mistakes:

• Expecting None if tasks run in order
• Confusing key names or task_ids
• Assuming pull returns error if no data

4. You tried to push a large file content directly into XCom but Airflow throws an error. What is the best way to fix this issue?

medium

A. Store the large file in external storage and push only its reference in XCom

B. Increase Airflow's XCom size limit in the configuration file

C. Split the large file into many small pieces and push each piece separately in XCom

D. Disable XCom size checks in Airflow's source code

Solution

1. Step 1: Identify the cause of the error

   Airflow limits XCom size; pushing large files directly causes errors.

2. Step 2: Choose the best practice alternative

   Storing large data externally and pushing only a reference keeps Airflow stable and avoids errors.

3. Final Answer:

   Store the large file in external storage and push only its reference in XCom -> Option A

4. Quick Check:

   Large data in XCom? Use external storage reference [OK]

Hint: Use external storage for large data, push reference only [OK]

Common Mistakes:

• Trying to increase XCom size limit (not supported)
• Splitting data manually is complex and error-prone
• Modifying Airflow source code is risky and unnecessary

5. You have a DAG where a task generates a large JSON report. You want to pass this report to the next task without slowing down Airflow. Which approach is best?

hard

A. Save the JSON report to an S3 bucket and push the S3 path via XCom

B. Serialize the JSON report and push it directly in XCom

C. Convert the JSON to a string and push it in XCom without size check

D. Store the JSON report in Airflow variables and read it in the next task

Solution

1. Step 1: Recognize XCom size limits

   Large JSON data should not be pushed directly to XCom to avoid performance issues.

2. Step 2: Select best alternative for large data sharing

   Saving the report externally (e.g., S3) and pushing only the path keeps Airflow fast and stable.

3. Final Answer:

   Save the JSON report to an S3 bucket and push the S3 path via XCom -> Option A

4. Quick Check:

   Large data? Use external storage + XCom reference [OK]

Hint: Push large data externally, pass only reference in XCom [OK]

Common Mistakes:

• Pushing large JSON directly in XCom
• Ignoring Airflow variables size limits
• Assuming string conversion bypasses size limits