How to Use Environment Variables in Airflow for Configuration

In Airflow, you can use os.environ to access environment variables inside your DAGs or tasks. Set environment variables in your system or Airflow configuration, then import os in your DAG file and use os.environ.get('VARIABLE_NAME') to retrieve their values.

Syntax

To use environment variables in Airflow, import the os module and access variables with os.environ.get('VARIABLE_NAME'). This returns the value of the environment variable or None if it is not set.

You can also provide a default value like os.environ.get('VARIABLE_NAME', 'default_value') to avoid errors if the variable is missing.

python

import os

my_var = os.environ.get('MY_ENV_VAR')
print(f"Value of MY_ENV_VAR: {my_var}")

Output

Value of MY_ENV_VAR: None

Example

This example shows how to use an environment variable inside an Airflow DAG to configure a task parameter dynamically.

python

import os
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_env_var():
    my_var = os.environ.get('MY_ENV_VAR', 'default_value')
    print(f"Environment variable MY_ENV_VAR is: {my_var}")

with DAG(dag_id='env_var_example', start_date=datetime(2024, 1, 1), schedule_interval='@once', catchup=False) as dag:
    task = PythonOperator(
        task_id='print_env_var_task',
        python_callable=print_env_var
    )

Output

Environment variable MY_ENV_VAR is: default_value

Common Pitfalls

• Not setting the environment variable: If the variable is not set in the system or Airflow environment, os.environ.get() returns None or the default value, which may cause unexpected behavior.
• Setting variables after Airflow starts: Airflow workers and scheduler read environment variables at startup, so changes require restarting Airflow components.
• Using os.environ['VAR'] without checking: This raises a KeyError if the variable is missing. Use os.environ.get() instead.

python

import os

# Wrong way - raises error if variable missing
# my_var = os.environ['MY_ENV_VAR']

# Right way - safe access with default
my_var = os.environ.get('MY_ENV_VAR', 'default_value')

Quick Reference

Here is a quick summary of how to work with environment variables in Airflow:

Action	Command/Code	Notes
Set environment variable in shell	export MY_ENV_VAR='value'	Linux/macOS terminal before starting Airflow
Access in Python code	os.environ.get('MY_ENV_VAR')	Returns value or None if missing
Access with default	os.environ.get('MY_ENV_VAR', 'default')	Avoids errors if variable not set
Avoid KeyError	Do not use os.environ['MY_ENV_VAR'] without check	Use get() method instead
Restart Airflow	Restart scheduler and workers after env changes	To apply new environment variables

Key Takeaways

Use os.environ.get('VAR_NAME') in Airflow DAGs to safely access environment variables.

Always set environment variables before starting Airflow components to ensure they are recognized.

Provide default values with os.environ.get() to avoid errors if variables are missing.

Avoid using os.environ['VAR_NAME'] directly to prevent KeyError exceptions.

Restart Airflow scheduler and workers after changing environment variables to apply updates.