Variable encryption for secrets in Apache Airflow - Commands & Configuration

Introduction

When you store sensitive information like passwords or API keys in Airflow variables, you want to keep them safe. Variable encryption helps protect these secrets so only authorized parts of your system can read them.

When you need to store database passwords securely in Airflow variables.

When your workflows require API keys that should not be visible in plain text.

When you want to prevent accidental exposure of secrets in logs or UI.

When multiple team members use Airflow but only some should access sensitive data.

When you want to comply with security policies requiring encrypted secrets.

Config File - airflow.cfg

airflow.cfg

[secrets]
enable_variable_encryption = True

[core]
fernet_key = 3x4mpl3F3rn3tK3y1234567890abcdefgHIJKLmnopqrs=

The enable_variable_encryption option turns on encryption for Airflow variables.

The fernet_key under the [core] section is the secret key used to encrypt and decrypt variables. It must be a valid Fernet key.

Commands

This command stores a variable named my_secret_password with the value supersecret123. Because encryption is enabled, the value is stored encrypted.

Terminal

airflow variables set my_secret_password supersecret123

Expected Output

No output (command runs silently)

This command retrieves the decrypted value of the variable my_secret_password. Airflow automatically decrypts it using the Fernet key.

Terminal

airflow variables get my_secret_password

Expected Output

supersecret123

This command lists all variables stored in Airflow. Encrypted values are not shown here, only variable names.

Terminal

airflow variables list

Expected Output

my_secret_password

Key Concept

If you remember nothing else from this pattern, remember: Airflow uses a Fernet key to encrypt variables so secrets stay safe and only decrypt when accessed with the right key.

Common Mistakes

Not setting a valid Fernet key in airflow.cfg before enabling variable encryption.

Without a valid Fernet key, Airflow cannot encrypt or decrypt variables, causing errors or storing secrets in plain text.

Generate a Fernet key using 'python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"' and set it in airflow.cfg under [core] as fernet_key.

Changing the Fernet key after variables are encrypted.

Changing the key breaks decryption of existing variables, making secrets inaccessible.

Keep the Fernet key constant or rotate it carefully using Airflow's key rotation procedures.

Storing secrets in variables without enabling encryption.

Secrets will be stored in plain text and visible to anyone with access to the Airflow metadata database or UI.

Always enable variable encryption in airflow.cfg before storing sensitive data.

Summary

Enable variable encryption by setting enable_variable_encryption = True and a valid fernet_key in airflow.cfg.

Use 'airflow variables set' to store secrets; they are encrypted automatically.

Retrieve secrets with 'airflow variables get' which decrypts them transparently.