XCom size limitations and alternatives in Apache Airflow - Mini Project: Build & Apply

XCom Size Limitations and Alternatives in Airflow

📖 Scenario: You are working with Apache Airflow to automate data workflows. You want to pass data between tasks using XComs, but you learn that XComs have size limits and storing large data directly can cause issues.This project will guide you through creating a simple Airflow DAG that demonstrates the size limitation of XComs and shows an alternative approach by storing large data in a file and passing only the file path via XCom.

🎯 Goal: Build an Airflow DAG with two tasks:A task that tries to push a large data object to XCom (which is not recommended).A task that pushes a file path to XCom instead, demonstrating a better alternative for large data.You will learn how to handle XCom size limits and use alternatives effectively.

📋 What You'll Learn

Create a Python dictionary with a large data string

Create a variable for the file path to store large data

Push large data directly to XCom in one task

Push file path to XCom in another task

Print the XCom values in the final step

💡 Why This Matters

🌍 Real World

In real Airflow workflows, passing large data directly via XCom can cause failures or slowdowns. Using file paths or external storage is a common practice.

💼 Career

Understanding XCom size limits and alternatives is important for building reliable and scalable data pipelines in Airflow, a key skill for DevOps and data engineering roles.

1

Create a large data dictionary

Create a Python dictionary called large_data with a key 'data' and a value that is a string of 10000 'A' characters.

Apache Airflow

# Create the large_data dictionary with 10000 'A' characters
# Your code here

Need a hint?

Use string multiplication like 'A' * 10000 to create a long string.

2

Create a file path variable for large data

Create a variable called file_path and set it to the string '/tmp/large_data.txt'.

Apache Airflow

large_data = {'data': 'A' * 10000}
# Create the file_path variable with '/tmp/large_data.txt'
# Your code here

Need a hint?

Just assign the string path to the variable file_path.

3

Push large data and file path to XCom in tasks

Write two Python functions: push_large_data and push_file_path. In push_large_data, push large_data to XCom using ti.xcom_push(key='large_data', value=large_data). In push_file_path, push file_path to XCom using ti.xcom_push(key='file_path', value=file_path).

Apache Airflow

large_data = {'data': 'A' * 10000}
file_path = '/tmp/large_data.txt'

# Define push_large_data function
# Your code here

# Define push_file_path function
# Your code here

Need a hint?

Use the ti parameter to push data to XCom with ti.xcom_push.

4

Print XCom values in a task

Write a function called print_xcom_values that pulls the large_data and file_path from XCom using ti.xcom_pull(key='large_data') and ti.xcom_pull(key='file_path'). Then print both values exactly as shown: print('Large data length:', len(large_data['data'])) and print('File path:', file_path).

Apache Airflow

large_data = {'data': 'A' * 10000}
file_path = '/tmp/large_data.txt'

def push_large_data(ti):
    ti.xcom_push(key='large_data', value=large_data)

def push_file_path(ti):
    ti.xcom_push(key='file_path', value=file_path)

# Define print_xcom_values function
# Your code here

Need a hint?

Use ti.xcom_pull to get values and print to show them.