Variable encryption for secrets in Apache Airflow - Step-by-Step Execution

Process Flow - Variable encryption for secrets

Define Secret Variable

↓

Encrypt Variable Value

↓

Store Encrypted Value in Airflow Metadata DB

↓

Retrieve Encrypted Variable

↓

Decrypt Variable Value at Runtime

↓

Use Decrypted Secret in DAG or Task

This flow shows how a secret variable is encrypted before storage and decrypted when used in Airflow.

Execution Sample

Apache Airflow

from airflow.models import Variable

# Set encrypted variable
Variable.set('db_password', 'MySecret123', encrypt=True)

# Get decrypted variable
password = Variable.get('db_password')

This code sets a secret variable with encryption and retrieves it decrypted for use.

Process Table

Step	Action	Input/Condition	Result/Output
1	Call Variable.set	key='db_password', value='MySecret123', encrypt=True	Value encrypted and stored in metadata DB
2	Store encrypted value	Encrypted data saved	Metadata DB updated with encrypted secret
3	Call Variable.get	key='db_password'	Encrypted value retrieved from DB
4	Decrypt value	Encrypted data from DB	Decrypted value 'MySecret123' returned
5	Use decrypted value	password variable assigned	password='MySecret123' ready for DAG use
6	End	No more actions	Secret variable securely handled

💡 All steps complete; secret variable encrypted at rest and decrypted on access

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
db_password (stored)	None	Encrypted data	Encrypted data	Encrypted data	Encrypted data
password (runtime)	None	None	MySecret123	MySecret123	MySecret123

Key Moments - 2 Insights

Why do we see encrypted data stored but decrypted data when we get the variable?

Because Airflow encrypts the variable before saving it to the database (step 1 and 2), but decrypts it automatically when you retrieve it (step 4), so you always get the secret in plain text at runtime.

What happens if we set encrypt=False when storing a secret?

The variable is stored as plain text in the metadata database, which is less secure. The execution_table shows encryption only when encrypt=True is used in step 1.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of 'password' after step 3?

A Encrypted data

B None

C MySecret123

D Error

💡 Hint

Check the 'password (runtime)' row in variable_tracker after step 3

At which step does Airflow decrypt the secret variable?

A Step 4

B Step 1

C Step 2

D Step 5

💡 Hint

Look at the 'Action' column in execution_table where decryption happens

If we set encrypt=False when storing, how would the stored value change in the execution table?

A Value stored would be encrypted

B Value stored would be plain text

C Value would not be stored

D Value would be deleted

💡 Hint

Refer to key_moments explanation about encrypt=False effect

Concept Snapshot

Airflow Variable Encryption:
- Use Variable.set(key, value, encrypt=True) to store secrets encrypted.
- Encrypted secrets saved in metadata DB.
- Variable.get(key) automatically decrypts secrets at runtime.
- Keeps secrets safe at rest and usable in DAGs.
- Avoid storing secrets without encryption.

Full Transcript

This visual execution shows how Airflow handles secret variables securely. First, when you set a variable with encrypt=True, Airflow encrypts the value before saving it in the metadata database. This protects the secret from being stored in plain text. Later, when you get the variable, Airflow retrieves the encrypted value and decrypts it automatically so your DAG or task can use the secret in plain text. The execution table traces each step from setting to using the secret, and the variable tracker shows how the stored and runtime variables change. Key moments clarify why encryption happens at storage and decryption at retrieval, and what happens if encryption is not used. The quiz tests understanding of these steps. This process ensures your secrets are safe at rest but accessible when needed.