XCom size limitations and alternatives in Apache Airflow - Commands & Configuration

Introduction

Airflow uses XComs to share small pieces of data between tasks. However, XComs have size limits because they store data in the Airflow database. This can cause problems when you try to pass large data. Knowing how to handle this helps keep your workflows smooth.

When you want to pass a small result like a file name or status message from one task to another.

When you try to share large data like big files or big JSON objects and get errors or slow performance.

When you want to keep your Airflow database fast and avoid it filling up with large data.

When you want to use external storage to share big data between tasks.

When you want to improve reliability by avoiding database timeouts caused by large XComs.

Commands

This command pulls the XCom value from the task 'example_task' in the DAG 'example_dag'. It shows the data passed between tasks.

Terminal

airflow tasks xcom_pull -d example_dag -t example_task

Expected Output

"small_result_value"

→

-d - Specifies the DAG id to pull XCom from

→

-t - Specifies the task id to pull XCom from

Runs the task 'example_task' locally to generate and push an XCom value. Useful to test XCom behavior.

Terminal

airflow tasks run example_dag example_task 2024-06-01T00:00:00 --local

Expected Output

[2024-06-01 00:00:00,000] {taskinstance.py:1234} INFO - Task succeeded

→

--local - Runs the task without scheduler for testing

Uploads a large data file to external storage (S3) to avoid storing it in XCom.

Terminal

aws s3 cp large_data.json s3://my-airflow-bucket/large_data.json

Expected Output

upload: ./large_data.json to s3://my-airflow-bucket/large_data.json

Pushes the S3 path as a small XCom value instead of the large data itself. Tasks can then read the file from S3.

Terminal

airflow tasks xcom_push --key large_data_path --value s3://my-airflow-bucket/large_data.json -d example_dag -t example_task --execution-date 2024-06-01T00:00:00

Expected Output

No output (command runs silently)

→

--key - Sets the XCom key name

→

--value - Sets the XCom value

→

-d - Specifies the DAG id

→

-t - Specifies the task id

→

--execution-date - Specifies the execution date of the task instance

Key Concept

If you remember nothing else, remember: XComs are for small data only; use external storage for large data and pass references via XCom.

Common Mistakes

Trying to push large files or big JSON objects directly into XCom.

This causes Airflow database to slow down or fail because XCom size is limited.

Store large data in external storage like S3 or a database, then push only the file path or reference as XCom.

Ignoring XCom size limits and expecting all data to pass smoothly.

Large data can cause task failures or long delays in Airflow UI and scheduler.

Design tasks to share only small metadata or pointers via XCom.

Summary

XComs in Airflow are designed to pass small pieces of data between tasks.

Large data should be stored outside Airflow, such as in cloud storage, and only references passed via XCom.

Use commands to push and pull XCom values carefully to avoid size limit issues.

Practice

1. What is the main reason to avoid storing large data directly in Airflow's XCom?

easy

A. Because XCom automatically deletes large data after one hour

B. Because Airflow does not support any data sharing between tasks

C. Because large data can slow down Airflow and cause stability issues

D. Because large data is encrypted and cannot be read by tasks

Solution

1. Step 1: Understand XCom's purpose

   XCom is designed for small data sharing between tasks, not large data storage.

2. Step 2: Recognize impact of large data in XCom

   Storing large data in XCom can slow down Airflow and cause stability problems.

3. Final Answer:

   Because large data can slow down Airflow and cause stability issues -> Option C

4. Quick Check:

   XCom size limits = Avoid large data [OK]

Hint: Remember: XCom is for small data only [OK]

Common Mistakes:

• Thinking XCom can handle large files safely
• Believing XCom auto-deletes large data quickly
• Assuming Airflow does not support task communication

2. Which of the following is the correct way to push a small value to XCom in an Airflow PythonOperator task?

easy

A. task_instance.push_xcom('result', 123)

B. task_instance.xcom_push(key='result', value=123)

C. xcom_push(task_instance, 'result', 123)

D. task_instance.xcom('result', 123)

Solution

1. Step 1: Recall XCom push method syntax

   The correct method to push data is task_instance.xcom_push(key, value).

2. Step 2: Match the correct syntax

   task_instance.xcom_push(key='result', value=123) uses the correct method and parameter names.

3. Final Answer:

   task_instance.xcom_push(key='result', value=123) -> Option B

4. Quick Check:

   XCom push method = xcom_push(key, value) [OK]

Hint: Use xcom_push with key and value parameters [OK]

Common Mistakes:

• Using incorrect method names like push_xcom
• Omitting key or value parameters
• Calling xcom as a method instead of xcom_push

3. Given the following code snippet in an Airflow task, what will be the output when retrieving the XCom value?

def push_func(ti):
    ti.xcom_push(key='data', value='small string')

def pull_func(ti):
    return ti.xcom_pull(key='data', task_ids='push_task')

# Assume push_func runs first and then pull_func runs
result = pull_func(ti)

medium

A. Empty string

B. None

C. KeyError

D. 'small string'

Solution

1. Step 1: Understand XCom push and pull sequence

   The first task pushes 'small string' with key 'data'. The second task pulls it using the same key and task id.

2. Step 2: Predict the pull result

   Since the key and task id match, the pull returns the pushed value 'small string'.

3. Final Answer:

   'small string' -> Option D

4. Quick Check:

   XCom pull returns pushed value if keys match [OK]

Hint: Pull returns pushed value if keys and task_ids match [OK]

Common Mistakes:

• Expecting None if tasks run in order
• Confusing key names or task_ids
• Assuming pull returns error if no data

4. You tried to push a large file content directly into XCom but Airflow throws an error. What is the best way to fix this issue?

medium

A. Store the large file in external storage and push only its reference in XCom

B. Increase Airflow's XCom size limit in the configuration file

C. Split the large file into many small pieces and push each piece separately in XCom

D. Disable XCom size checks in Airflow's source code

Solution

1. Step 1: Identify the cause of the error

   Airflow limits XCom size; pushing large files directly causes errors.

2. Step 2: Choose the best practice alternative

   Storing large data externally and pushing only a reference keeps Airflow stable and avoids errors.

3. Final Answer:

   Store the large file in external storage and push only its reference in XCom -> Option A

4. Quick Check:

   Large data in XCom? Use external storage reference [OK]

Hint: Use external storage for large data, push reference only [OK]

Common Mistakes:

• Trying to increase XCom size limit (not supported)
• Splitting data manually is complex and error-prone
• Modifying Airflow source code is risky and unnecessary

5. You have a DAG where a task generates a large JSON report. You want to pass this report to the next task without slowing down Airflow. Which approach is best?

hard

A. Save the JSON report to an S3 bucket and push the S3 path via XCom

B. Serialize the JSON report and push it directly in XCom

C. Convert the JSON to a string and push it in XCom without size check

D. Store the JSON report in Airflow variables and read it in the next task

Solution

1. Step 1: Recognize XCom size limits

   Large JSON data should not be pushed directly to XCom to avoid performance issues.

2. Step 2: Select best alternative for large data sharing

   Saving the report externally (e.g., S3) and pushing only the path keeps Airflow fast and stable.

3. Final Answer:

   Save the JSON report to an S3 bucket and push the S3 path via XCom -> Option A

4. Quick Check:

   Large data? Use external storage + XCom reference [OK]

Hint: Push large data externally, pass only reference in XCom [OK]

Common Mistakes:

• Pushing large JSON directly in XCom
• Ignoring Airflow variables size limits
• Assuming string conversion bypasses size limits