How to Use Variable.set in Airflow for Dynamic Configuration

Use Variable.set(key, value) in Airflow to store or update a variable with a specific key and value. This allows you to save configuration or data that can be accessed across DAGs and tasks dynamically.

Syntax

The Variable.set method requires two main arguments: key and value. The key is a string that identifies the variable, and the value is the data you want to store, which can be a string or JSON-serializable object.

Optionally, you can use the serialize_json=True parameter to store complex data structures as JSON.

python

from airflow.models import Variable

Variable.set(key, value, serialize_json=False)

Example

This example shows how to set a simple string variable and a JSON variable in Airflow using Variable.set. The variables can then be accessed later in your DAG or tasks.

python

from airflow.models import Variable

# Set a simple string variable
Variable.set("my_var", "hello world")

# Set a JSON-serializable variable
data = {"key1": "value1", "key2": 2}
Variable.set("my_json_var", data, serialize_json=True)

# Retrieve and print variables
print(Variable.get("my_var"))
print(Variable.get("my_json_var", deserialize_json=True))

Output

hello world
{'key1': 'value1', 'key2': 2}

Common Pitfalls

• Forgetting to import Variable from airflow.models.
• Not using serialize_json=True when storing complex data, which causes data to be saved as a string instead of JSON.
• Trying to set variables inside tasks without proper context or permissions.
• Overwriting important variables unintentionally by using the same key.

python

from airflow.models import Variable

# Wrong: storing dict without serialize_json
Variable.set("wrong_var", {"a": 1, "b": 2})  # Stored as string representation

# Right: store dict as JSON
Variable.set("right_var", {"a": 1, "b": 2}, serialize_json=True)

Quick Reference

Use this quick reference to remember how to set and get variables in Airflow:

Action	Code Example	Description
Set string variable	Variable.set("key", "value")	Stores a simple string value.
Set JSON variable	Variable.set("key", data, serialize_json=True)	Stores complex data as JSON.
Get string variable	Variable.get("key")	Retrieves stored string value.
Get JSON variable	Variable.get("key", deserialize_json=True)	Retrieves stored JSON data as Python object.

Key Takeaways

Use Variable.set(key, value) to store or update Airflow variables.

Set serialize_json=True to save complex data structures as JSON.

Always import Variable from airflow.models before using it.

Avoid overwriting variables unintentionally by using unique keys.

Retrieve variables with Variable.get using deserialize_json=True for JSON data.