What is Variable in Airflow: Definition and Usage

In Apache Airflow, a Variable is a way to store and retrieve small pieces of data or configuration as key-value pairs. These variables help you manage dynamic values that your workflows (DAGs) can use at runtime without changing the code.

How It Works

Think of Airflow Variables like sticky notes you keep on your desk. Each note has a label (key) and some information (value). When your workflow runs, it can look at these notes to get the information it needs without hardcoding it inside the workflow.

Variables are stored centrally in Airflow's metadata database. You can create, update, or delete them through the Airflow web interface or programmatically. When a workflow runs, it asks Airflow for the value of a variable by its key, and Airflow returns the stored value.

This system lets you change configuration or parameters on the fly without touching your workflow code, making your workflows more flexible and easier to maintain.

Example

This example shows how to set and get a variable in an Airflow DAG using Python code.

python

from airflow.models import Variable

# Set a variable (usually done once, not inside DAG code)
Variable.set("greeting", "Hello, Airflow!")

# Get the variable value inside a DAG or task
message = Variable.get("greeting")
print(message)

Output

Hello, Airflow!

When to Use

Use Airflow Variables when you want to store configuration values or parameters that might change over time or between environments (like development, testing, production).

For example, you can store API keys, file paths, email addresses, or feature flags as variables. This way, you can update these values without changing your DAG code or redeploying workflows.

Variables are especially useful when you want to share settings across multiple workflows or tasks, or when you want to keep sensitive information out of your codebase.

Key Points

• Variables store key-value pairs accessible by all DAGs.
• They help separate configuration from code.
• Variables can be managed via Airflow UI or code.
• Use them to keep workflows flexible and maintainable.
• Be cautious with sensitive data; consider Airflow Connections or secrets backend for secrets.

Key Takeaways

Airflow Variables store dynamic key-value data accessible in workflows.

They let you change configuration without modifying DAG code.

Variables are managed centrally via Airflow UI or programmatically.

Use Variables for non-sensitive config; use secrets backend for sensitive info.

Variables improve workflow flexibility and maintainability.