XCom size limitations and alternatives in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the default size limit for an XCom value in Airflow?

The default size limit for an XCom value is about 48 KB because XComs are stored in the Airflow metadata database, which is not designed for large data.

beginner

Why should you avoid storing large data directly in XComs?

Storing large data in XComs can slow down the Airflow database, cause performance issues, and may lead to failures due to size limits.

beginner

Name one common alternative to storing large data instead of using XComs.

A common alternative is to store large data in external storage like Amazon S3, Google Cloud Storage, or a database, and pass only the reference or path via XCom.

intermediate

How can you pass data between tasks without hitting XCom size limits?

You can save the data to a file or cloud storage in one task, then pass the file path or URL via XCom to the next task, which reads the data from there.

advanced

What is the role of XCom backends in managing XCom size limitations?

Custom XCom backends can be implemented to store XCom data outside the metadata database, such as in files or object storage, helping to handle larger data sizes.

What happens if you try to store very large data directly in an XCom?

A It may cause database performance issues or errors due to size limits.

B Airflow automatically compresses the data without issues.

C The data is split into multiple XCom entries automatically.

D Airflow stores it in a temporary file without user intervention.

XComs are stored in the Airflow metadata database, which has size limits. Large data can cause performance problems or errors.

Which of the following is a good practice to handle large data between Airflow tasks?

A Use environment variables to pass the data.

B Encode the data as a very long string in XCom.

C Send the data via email between tasks.

D Store the data in cloud storage and pass the location via XCom.

Storing large data externally and passing only the reference via XCom avoids size limits and improves performance.

What is an XCom backend in Airflow?

A A security feature to encrypt XCom data.

B A plugin to customize how and where XCom data is stored.

C A scheduler for XCom data cleanup.

D A tool to visualize XCom data in the UI.

XCom backends allow customizing storage of XCom data, for example, storing large data outside the metadata database.

What type of data is best suited for XComs?

A Large datasets or files.

B Binary executable files.

C Small metadata or references like file paths or IDs.

D Encrypted passwords.

XComs are designed for small pieces of data like metadata or references, not large files or datasets.

If you want to pass a large JSON object between tasks, what should you do?

A Save the JSON to external storage and pass the path via XCom.

B Directly push the JSON object to XCom without changes.

C Convert the JSON to XML and store in XCom.

D Split the JSON into multiple XComs automatically.

Large JSON objects should be stored externally and only the reference passed via XCom to avoid size limits.

Explain why XComs have size limitations and how you can work around them in Airflow.

Think about where XCom data is stored and what happens with big data.

Describe how to implement an alternative to large XCom data passing using cloud storage.

Consider splitting data storage and data reference.

Practice

1. What is the main reason to avoid storing large data directly in Airflow's XCom?

easy

A. Because XCom automatically deletes large data after one hour

B. Because Airflow does not support any data sharing between tasks

C. Because large data can slow down Airflow and cause stability issues

D. Because large data is encrypted and cannot be read by tasks

Solution

1. Step 1: Understand XCom's purpose

   XCom is designed for small data sharing between tasks, not large data storage.

2. Step 2: Recognize impact of large data in XCom

   Storing large data in XCom can slow down Airflow and cause stability problems.

3. Final Answer:

   Because large data can slow down Airflow and cause stability issues -> Option C

4. Quick Check:

   XCom size limits = Avoid large data [OK]

Hint: Remember: XCom is for small data only [OK]

Common Mistakes:

• Thinking XCom can handle large files safely
• Believing XCom auto-deletes large data quickly
• Assuming Airflow does not support task communication

2. Which of the following is the correct way to push a small value to XCom in an Airflow PythonOperator task?

easy

A. task_instance.push_xcom('result', 123)

B. task_instance.xcom_push(key='result', value=123)

C. xcom_push(task_instance, 'result', 123)

D. task_instance.xcom('result', 123)

Solution

1. Step 1: Recall XCom push method syntax

   The correct method to push data is task_instance.xcom_push(key, value).

2. Step 2: Match the correct syntax

   task_instance.xcom_push(key='result', value=123) uses the correct method and parameter names.

3. Final Answer:

   task_instance.xcom_push(key='result', value=123) -> Option B

4. Quick Check:

   XCom push method = xcom_push(key, value) [OK]

Hint: Use xcom_push with key and value parameters [OK]

Common Mistakes:

• Using incorrect method names like push_xcom
• Omitting key or value parameters
• Calling xcom as a method instead of xcom_push

3. Given the following code snippet in an Airflow task, what will be the output when retrieving the XCom value?

def push_func(ti):
    ti.xcom_push(key='data', value='small string')

def pull_func(ti):
    return ti.xcom_pull(key='data', task_ids='push_task')

# Assume push_func runs first and then pull_func runs
result = pull_func(ti)

medium

A. Empty string

B. None

C. KeyError

D. 'small string'

Solution

1. Step 1: Understand XCom push and pull sequence

   The first task pushes 'small string' with key 'data'. The second task pulls it using the same key and task id.

2. Step 2: Predict the pull result

   Since the key and task id match, the pull returns the pushed value 'small string'.

3. Final Answer:

   'small string' -> Option D

4. Quick Check:

   XCom pull returns pushed value if keys match [OK]

Hint: Pull returns pushed value if keys and task_ids match [OK]

Common Mistakes:

• Expecting None if tasks run in order
• Confusing key names or task_ids
• Assuming pull returns error if no data

4. You tried to push a large file content directly into XCom but Airflow throws an error. What is the best way to fix this issue?

medium

A. Store the large file in external storage and push only its reference in XCom

B. Increase Airflow's XCom size limit in the configuration file

C. Split the large file into many small pieces and push each piece separately in XCom

D. Disable XCom size checks in Airflow's source code

Solution

1. Step 1: Identify the cause of the error

   Airflow limits XCom size; pushing large files directly causes errors.

2. Step 2: Choose the best practice alternative

   Storing large data externally and pushing only a reference keeps Airflow stable and avoids errors.

3. Final Answer:

   Store the large file in external storage and push only its reference in XCom -> Option A

4. Quick Check:

   Large data in XCom? Use external storage reference [OK]

Hint: Use external storage for large data, push reference only [OK]

Common Mistakes:

• Trying to increase XCom size limit (not supported)
• Splitting data manually is complex and error-prone
• Modifying Airflow source code is risky and unnecessary

5. You have a DAG where a task generates a large JSON report. You want to pass this report to the next task without slowing down Airflow. Which approach is best?

hard

A. Save the JSON report to an S3 bucket and push the S3 path via XCom

B. Serialize the JSON report and push it directly in XCom

C. Convert the JSON to a string and push it in XCom without size check

D. Store the JSON report in Airflow variables and read it in the next task

Solution

1. Step 1: Recognize XCom size limits

   Large JSON data should not be pushed directly to XCom to avoid performance issues.

2. Step 2: Select best alternative for large data sharing

   Saving the report externally (e.g., S3) and pushing only the path keeps Airflow fast and stable.

3. Final Answer:

   Save the JSON report to an S3 bucket and push the S3 path via XCom -> Option A

4. Quick Check:

   Large data? Use external storage + XCom reference [OK]

Hint: Push large data externally, pass only reference in XCom [OK]

Common Mistakes:

• Pushing large JSON directly in XCom
• Ignoring Airflow variables size limits
• Assuming string conversion bypasses size limits