Variable encryption for secrets in Apache Airflow - Mini Project: Build & Apply

Variable encryption for secrets

📖 Scenario: You are managing an Apache Airflow environment where you need to store sensitive information like API keys securely. Airflow allows you to store variables, but storing secrets in plain text is risky. To protect these secrets, Airflow supports encrypting variables.In this project, you will simulate storing a secret variable in Airflow with encryption enabled, and then retrieve and decrypt it safely.

🎯 Goal: Build a simple Python script that simulates storing an encrypted secret variable and retrieving it decrypted, mimicking Airflow's variable encryption feature.

📋 What You'll Learn

Create a dictionary to simulate Airflow variables storage

Add a configuration variable to represent encryption key

Write a function to encrypt and decrypt variable values

Store an encrypted secret and retrieve it decrypted

💡 Why This Matters

🌍 Real World

In real Airflow deployments, secrets like API keys and passwords must be encrypted to prevent unauthorized access. This project shows the basic idea behind encrypting variables before storing them.

💼 Career

Understanding how to securely manage secrets is essential for DevOps engineers and data engineers working with Airflow or any automation platform that handles sensitive data.

1

Create Airflow variables storage

Create a dictionary called airflow_variables with one entry: the key 'api_key' and the value 'my_secret_api_key'.

Apache Airflow

# Create the airflow_variables dictionary with the specified entry
# Your code here

Hint

Use curly braces to create a dictionary with the key 'api_key' and the value 'my_secret_api_key'.

2

Add encryption key configuration

Create a variable called encryption_key and set it to the string 'simplekey'.

Apache Airflow

airflow_variables = {'api_key': 'my_secret_api_key'}
# Create the encryption_key variable with the value 'simplekey'
# Your code here

Hint

Just assign the string 'simplekey' to the variable encryption_key.

3

Write encryption and decryption functions

Define two functions: encrypt(value, key) and decrypt(value, key). Use a simple Caesar cipher shifting each character by the length of key. For encryption, shift characters forward; for decryption, shift backward.

Apache Airflow

airflow_variables = {'api_key': 'my_secret_api_key'}
encryption_key = 'simplekey'
# Define encrypt(value, key) and decrypt(value, key) functions
# Your code here

Hint

Use ord() and chr() to shift characters by the length of the key.

4

Store encrypted secret and print decrypted value

Encrypt the 'api_key' value from airflow_variables using encrypt and encryption_key. Store it back in airflow_variables['api_key']. Then decrypt it and print the decrypted value.

Apache Airflow

airflow_variables = {'api_key': 'my_secret_api_key'}
encryption_key = 'simplekey'

def encrypt(value, key):
    shift = len(key)
    encrypted = ''.join(chr((ord(c) + shift) % 256) for c in value)
    return encrypted

def decrypt(value, key):
    shift = len(key)
    decrypted = ''.join(chr((ord(c) - shift) % 256) for c in value)
    return decrypted

# Encrypt the api_key, store it, then decrypt and print it
# Your code here

Hint

Use the encrypt function on the original value, store it, then decrypt and print the result.

Practice

1. What is the main purpose of encrypting variables in Airflow?

easy

A. To keep sensitive data like passwords safe from unauthorized access

B. To speed up the execution of DAGs

C. To reduce the size of variable storage

D. To make variables visible in the Airflow UI

Solution

1. Step 1: Understand variable encryption purpose

   Encryption protects sensitive information by making it unreadable without a key.

2. Step 2: Relate to Airflow variables

   Airflow encrypts variables to keep secrets like passwords safe from unauthorized users.

3. Final Answer:

   To keep sensitive data like passwords safe from unauthorized access -> Option A

4. Quick Check:

   Encryption = Protect secrets [OK]

Hint: Encryption means hiding secrets safely [OK]

Common Mistakes:

• Thinking encryption speeds up DAGs
• Confusing encryption with compression
• Believing encryption makes variables visible

2. Which command correctly sets an encrypted variable named API_KEY with value 12345 in Airflow CLI?

easy

A. airflow variables set --encrypt API_KEY 12345

B. airflow variables set API_KEY --encrypt 12345

C. airflow variables set API_KEY 12345 --encrypt

D. airflow variables set 12345 API_KEY --encrypt

Solution

1. Step 1: Recall Airflow CLI syntax for encrypted variables

   The correct syntax places the --encrypt flag before the variable name and value.

2. Step 2: Match syntax with options

   airflow variables set --encrypt API_KEY 12345 uses airflow variables set --encrypt API_KEY 12345, which is correct.

3. Final Answer:

   airflow variables set --encrypt API_KEY 12345 -> Option A

4. Quick Check:

   Flag before name and value = airflow variables set --encrypt API_KEY 12345 [OK]

Hint: Put --encrypt right after 'set' command [OK]

Common Mistakes:

• Placing --encrypt after variable name or value
• Swapping variable name and value order
• Omitting --encrypt flag

3. Given the following code snippet in an Airflow DAG:

from airflow.models import Variable
secret = Variable.get('MY_SECRET')

What will secret contain if MY_SECRET was set with encryption?

medium

A. The encrypted string stored in Airflow

B. The decrypted original secret value

C. An error because encrypted variables cannot be read

D. None, because encrypted variables are hidden

Solution

1. Step 1: Understand Variable.get behavior with encrypted variables

   Airflow automatically decrypts encrypted variables when accessed via Variable.get.

2. Step 2: Determine what secret holds

   It holds the original decrypted secret value, not the encrypted string or None.

3. Final Answer:

   The decrypted original secret value -> Option B

4. Quick Check:

   Variable.get decrypts automatically [OK]

Hint: Variable.get returns decrypted value automatically [OK]

Common Mistakes:

• Expecting encrypted string instead of decrypted
• Assuming Variable.get raises error for encrypted vars
• Thinking encrypted vars are hidden and return None

4. You tried to set an encrypted variable with this command:
airflow variables set API_SECRET 98765 --encrypt
But the variable is not encrypted. What is the likely mistake?

medium

A. The value must be base64 encoded before setting

B. The variable name cannot contain underscores

C. You must restart Airflow for encryption to work

D. The --encrypt flag should come before the variable name and value

Solution

1. Step 1: Check correct placement of --encrypt flag

   The flag must be placed immediately after set, before variable name and value.

2. Step 2: Analyze the command given

   The flag is placed after value, so encryption is not applied.

3. Final Answer:

   The --encrypt flag should come before the variable name and value -> Option D

4. Quick Check:

   Flag position matters for encryption [OK]

Hint: Place --encrypt right after 'set' command [OK]

Common Mistakes:

• Putting --encrypt after value
• Restarting Airflow unnecessarily
• Encoding value manually before setting

5. You want to securely store multiple secrets as Airflow variables encrypted at once. Which approach is best?

hard

A. Store all secrets in a plain text file and upload it to Airflow UI

B. Set variables without encryption and rely on OS file permissions

C. Write a script that uses airflow variables set --encrypt for each secret

D. Use Airflow connections instead of variables for secrets

Solution

1. Step 1: Understand best practice for multiple encrypted variables

   Using a script automates setting each secret with encryption flag, ensuring safety.

2. Step 2: Evaluate other options

   Uploading plain text or skipping encryption risks exposure; connections are different concept.

3. Final Answer:

   Write a script that uses airflow variables set --encrypt for each secret -> Option C

4. Quick Check:

   Automate encrypted variable setting for multiple secrets [OK]

Hint: Automate with script using --encrypt flag [OK]

Common Mistakes:

• Uploading plain text files
• Ignoring encryption for secrets
• Confusing variables with connections