Variable encryption for secrets in Apache Airflow - Step-by-Step Execution

Process Flow - Variable encryption for secrets

Define Secret Variable

↓

Encrypt Variable Value

↓

Store Encrypted Value in Airflow Metadata DB

↓

Retrieve Encrypted Variable

↓

Decrypt Variable Value at Runtime

↓

Use Decrypted Secret in DAG or Task

This flow shows how a secret variable is encrypted before storage and decrypted when used in Airflow.

Execution Sample

Apache Airflow

from airflow.models import Variable

# Set encrypted variable
Variable.set('db_password', 'MySecret123', encrypt=True)

# Get decrypted variable
password = Variable.get('db_password')

This code sets a secret variable with encryption and retrieves it decrypted for use.

Process Table

Step	Action	Input/Condition	Result/Output
1	Call Variable.set	key='db_password', value='MySecret123', encrypt=True	Value encrypted and stored in metadata DB
2	Store encrypted value	Encrypted data saved	Metadata DB updated with encrypted secret
3	Call Variable.get	key='db_password'	Encrypted value retrieved from DB
4	Decrypt value	Encrypted data from DB	Decrypted value 'MySecret123' returned
5	Use decrypted value	password variable assigned	password='MySecret123' ready for DAG use
6	End	No more actions	Secret variable securely handled

💡 All steps complete; secret variable encrypted at rest and decrypted on access

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
db_password (stored)	None	Encrypted data	Encrypted data	Encrypted data	Encrypted data
password (runtime)	None	None	MySecret123	MySecret123	MySecret123

Key Moments - 2 Insights

Why do we see encrypted data stored but decrypted data when we get the variable?

Because Airflow encrypts the variable before saving it to the database (step 1 and 2), but decrypts it automatically when you retrieve it (step 4), so you always get the secret in plain text at runtime.

What happens if we set encrypt=False when storing a secret?

The variable is stored as plain text in the metadata database, which is less secure. The execution_table shows encryption only when encrypt=True is used in step 1.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of 'password' after step 3?

A Encrypted data

B None

C MySecret123

D Error

💡 Hint

Check the 'password (runtime)' row in variable_tracker after step 3

At which step does Airflow decrypt the secret variable?

A Step 4

B Step 1

C Step 2

D Step 5

💡 Hint

Look at the 'Action' column in execution_table where decryption happens

If we set encrypt=False when storing, how would the stored value change in the execution table?

A Value stored would be encrypted

B Value stored would be plain text

C Value would not be stored

D Value would be deleted

💡 Hint

Refer to key_moments explanation about encrypt=False effect

Concept Snapshot

Airflow Variable Encryption:
- Use Variable.set(key, value, encrypt=True) to store secrets encrypted.
- Encrypted secrets saved in metadata DB.
- Variable.get(key) automatically decrypts secrets at runtime.
- Keeps secrets safe at rest and usable in DAGs.
- Avoid storing secrets without encryption.

Full Transcript

This visual execution shows how Airflow handles secret variables securely. First, when you set a variable with encrypt=True, Airflow encrypts the value before saving it in the metadata database. This protects the secret from being stored in plain text. Later, when you get the variable, Airflow retrieves the encrypted value and decrypts it automatically so your DAG or task can use the secret in plain text. The execution table traces each step from setting to using the secret, and the variable tracker shows how the stored and runtime variables change. Key moments clarify why encryption happens at storage and decryption at retrieval, and what happens if encryption is not used. The quiz tests understanding of these steps. This process ensures your secrets are safe at rest but accessible when needed.

Practice

1. What is the main purpose of encrypting variables in Airflow?

easy

A. To keep sensitive data like passwords safe from unauthorized access

B. To speed up the execution of DAGs

C. To reduce the size of variable storage

D. To make variables visible in the Airflow UI

Solution

1. Step 1: Understand variable encryption purpose

   Encryption protects sensitive information by making it unreadable without a key.

2. Step 2: Relate to Airflow variables

   Airflow encrypts variables to keep secrets like passwords safe from unauthorized users.

3. Final Answer:

   To keep sensitive data like passwords safe from unauthorized access -> Option A

4. Quick Check:

   Encryption = Protect secrets [OK]

Hint: Encryption means hiding secrets safely [OK]

Common Mistakes:

• Thinking encryption speeds up DAGs
• Confusing encryption with compression
• Believing encryption makes variables visible

2. Which command correctly sets an encrypted variable named API_KEY with value 12345 in Airflow CLI?

easy

A. airflow variables set --encrypt API_KEY 12345

B. airflow variables set API_KEY --encrypt 12345

C. airflow variables set API_KEY 12345 --encrypt

D. airflow variables set 12345 API_KEY --encrypt

Solution

1. Step 1: Recall Airflow CLI syntax for encrypted variables

   The correct syntax places the --encrypt flag before the variable name and value.

2. Step 2: Match syntax with options

   airflow variables set --encrypt API_KEY 12345 uses airflow variables set --encrypt API_KEY 12345, which is correct.

3. Final Answer:

   airflow variables set --encrypt API_KEY 12345 -> Option A

4. Quick Check:

   Flag before name and value = airflow variables set --encrypt API_KEY 12345 [OK]

Hint: Put --encrypt right after 'set' command [OK]

Common Mistakes:

• Placing --encrypt after variable name or value
• Swapping variable name and value order
• Omitting --encrypt flag

3. Given the following code snippet in an Airflow DAG:

from airflow.models import Variable
secret = Variable.get('MY_SECRET')

What will secret contain if MY_SECRET was set with encryption?

medium

A. The encrypted string stored in Airflow

B. The decrypted original secret value

C. An error because encrypted variables cannot be read

D. None, because encrypted variables are hidden

Solution

1. Step 1: Understand Variable.get behavior with encrypted variables

   Airflow automatically decrypts encrypted variables when accessed via Variable.get.

2. Step 2: Determine what secret holds

   It holds the original decrypted secret value, not the encrypted string or None.

3. Final Answer:

   The decrypted original secret value -> Option B

4. Quick Check:

   Variable.get decrypts automatically [OK]

Hint: Variable.get returns decrypted value automatically [OK]

Common Mistakes:

• Expecting encrypted string instead of decrypted
• Assuming Variable.get raises error for encrypted vars
• Thinking encrypted vars are hidden and return None

4. You tried to set an encrypted variable with this command:
airflow variables set API_SECRET 98765 --encrypt
But the variable is not encrypted. What is the likely mistake?

medium

A. The value must be base64 encoded before setting

B. The variable name cannot contain underscores

C. You must restart Airflow for encryption to work

D. The --encrypt flag should come before the variable name and value

Solution

1. Step 1: Check correct placement of --encrypt flag

   The flag must be placed immediately after set, before variable name and value.

2. Step 2: Analyze the command given

   The flag is placed after value, so encryption is not applied.

3. Final Answer:

   The --encrypt flag should come before the variable name and value -> Option D

4. Quick Check:

   Flag position matters for encryption [OK]

Hint: Place --encrypt right after 'set' command [OK]

Common Mistakes:

• Putting --encrypt after value
• Restarting Airflow unnecessarily
• Encoding value manually before setting

5. You want to securely store multiple secrets as Airflow variables encrypted at once. Which approach is best?

hard

A. Store all secrets in a plain text file and upload it to Airflow UI

B. Set variables without encryption and rely on OS file permissions

C. Write a script that uses airflow variables set --encrypt for each secret

D. Use Airflow connections instead of variables for secrets

Solution

1. Step 1: Understand best practice for multiple encrypted variables

   Using a script automates setting each secret with encryption flag, ensuring safety.

2. Step 2: Evaluate other options

   Uploading plain text or skipping encryption risks exposure; connections are different concept.

3. Final Answer:

   Write a script that uses airflow variables set --encrypt for each secret -> Option C

4. Quick Check:

   Automate encrypted variable setting for multiple secrets [OK]

Hint: Automate with script using --encrypt flag [OK]

Common Mistakes:

• Uploading plain text files
• Ignoring encryption for secrets
• Confusing variables with connections