Variable vs XCom in Airflow: Key Differences and Usage

In Airflow, Variables store global, persistent key-value pairs accessible across DAGs, while XComs pass small messages or data between tasks within the same DAG run. Variables are for configuration or shared data, and XComs are for task-to-task communication during execution.

Quick Comparison

This table summarizes the main differences between Airflow Variables and XComs.

Factor	Variables	XComs
Purpose	Store global configuration or shared data	Pass messages or small data between tasks
Scope	Global across all DAGs	Limited to a single DAG run
Persistence	Persistent until manually changed or deleted	Temporary, tied to task instance and DAG run
Data Size	Can store larger data (JSON, strings)	Best for small data (strings, numbers, small JSON)
Access Method	Accessed via Airflow UI, CLI, or API	Accessed programmatically in tasks
Use Case	Store credentials, configs, flags	Share task results or status

Key Differences

Variables in Airflow act like global settings or shared storage. They hold key-value pairs that any DAG or task can read or update. This makes them ideal for storing credentials, configuration flags, or any data that should persist beyond a single workflow run. Variables are accessible through the Airflow UI, CLI, or code, and changes affect all DAGs immediately.

XComs (short for "cross-communication") are designed for passing small pieces of data between tasks within the same DAG run. They are temporary and scoped to the specific execution of a DAG. XComs let tasks share results or signals dynamically during workflow execution, but they are not meant for large data or long-term storage.

In summary, use Variables for persistent, global data and XComs for ephemeral, task-to-task communication within a DAG run.

Code Comparison

Example showing how to use Variables to store and retrieve a value in Airflow tasks.

python

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.models import Variable
from datetime import datetime

def print_variable():
    my_var = Variable.get("my_key", default_var="default_value")
    print(f"Variable value: {my_var}")

def set_variable():
    Variable.set("my_key", "hello from variable")

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG('variable_example', default_args=default_args, schedule_interval=None)

set_var_task = PythonOperator(task_id='set_var', python_callable=set_variable, dag=dag)
print_var_task = PythonOperator(task_id='print_var', python_callable=print_variable, dag=dag)

set_var_task >> print_var_task

Output

Variable value: hello from variable

XCom Equivalent

Example showing how to use XComs to pass data between tasks in the same DAG run.

python

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def push_xcom(ti):
    ti.xcom_push(key='message', value='hello from xcom')

def pull_xcom(ti):
    msg = ti.xcom_pull(key='message', task_ids='push_task')
    print(f"XCom message: {msg}")

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG('xcom_example', default_args=default_args, schedule_interval=None)

push_task = PythonOperator(task_id='push_task', python_callable=push_xcom, dag=dag)
pull_task = PythonOperator(task_id='pull_task', python_callable=pull_xcom, dag=dag)

push_task >> pull_task

Output

XCom message: hello from xcom

When to Use Which

Choose Variables when you need persistent, global data accessible by multiple DAGs or tasks over time, such as configuration settings or credentials. Variables are best for data that does not change often during DAG runs and must be centrally managed.

Choose XComs when you need to pass small, temporary data between tasks within the same DAG run, like task results or status flags. XComs are ideal for dynamic communication during workflow execution and should not be used for large or persistent data.

Key Takeaways

Variables store persistent, global key-value data accessible across DAGs.

XComs pass small, temporary messages between tasks in the same DAG run.

Use Variables for configs and credentials; use XComs for task communication.

Variables persist until changed; XComs exist only during a DAG run.

XComs are not suitable for large data or cross-DAG sharing.