Return Value as XCom in Airflow: What It Is and How It Works

In Airflow, the return value as XCom means that the output of a task's Python function is automatically saved as an XCom (cross-communication) message. This allows other tasks to retrieve that value later, enabling simple data sharing between tasks without extra code.

How It Works

Think of Airflow tasks as workers on an assembly line. Sometimes, one worker finishes a part and needs to pass it to the next worker. In Airflow, this passing of data is done using XComs, which stands for cross-communication messages.

When a Python task returns a value, Airflow automatically stores this value as an XCom linked to that task instance. Other tasks can then ask Airflow to fetch this stored value to use it in their own work. This is like leaving a note on a shared bulletin board for the next worker to read.

This mechanism helps tasks share small pieces of data easily without needing complex storage or databases. It works behind the scenes, so you just return a value, and Airflow handles the rest.

Example

This example shows a simple Python task that returns a value, which Airflow saves as an XCom. Another task then pulls this value and prints it.

python

from airflow import DAG
from airflow.decorators import task
from airflow.utils.dates import days_ago

with DAG(dag_id='xcom_return_example', start_date=days_ago(1), schedule_interval=None) as dag:

    @task
    def push_value():
        return 'Hello from XCom!'

    @task
    def pull_value(ti=None):
        message = ti.xcom_pull(task_ids='push_value')
        print(f'Received message: {message}')

    push = push_value()
    pull = pull_value()
    push >> pull

Output

Received message: Hello from XCom!

When to Use

Use return value as XCom when you want to pass small data like strings, numbers, or small dictionaries between tasks in the same DAG run. It is perfect for sharing results, flags, or parameters without external storage.

For example, you might have a task that fetches data from an API and returns a summary count. The next task can use that count to decide what to do next. This avoids writing to files or databases just to share simple info.

However, avoid using XComs for large data or files, as they are stored in Airflow's metadata database and can slow down your system.

Key Points

• Returning a value from a Python task automatically saves it as an XCom.
• Other tasks can retrieve this value using ti.xcom_pull().
• XComs are best for small data sharing within the same DAG run.
• Do not use XComs for large files or big data.
• XComs simplify communication between tasks without extra storage setup.

Key Takeaways

Returning a value from a PythonOperator or @task saves it as an XCom automatically.

XComs enable easy data sharing between tasks in the same DAG run.

Use XComs only for small data to avoid performance issues.

Retrieve XCom values using task instance's xcom_pull method.

XComs simplify task communication without external storage.