Secrets management in Apache Airflow - Mini Project: Build & Apply

Secrets Management in Apache Airflow

📖 Scenario: You are setting up Apache Airflow to securely manage sensitive information like database passwords and API keys. Instead of hardcoding secrets in your DAGs, you will use Airflow's built-in secrets backend to fetch secrets at runtime.

🎯 Goal: Build a simple Airflow DAG that retrieves a secret value from Airflow's Variable store using the secrets backend and prints it. This will demonstrate how to manage secrets securely in Airflow.

📋 What You'll Learn

Create a dictionary to simulate Airflow Variables with secret values

Add a configuration variable to specify the secret key to fetch

Write a function to retrieve the secret from the dictionary using the key

Print the retrieved secret value

💡 Why This Matters

🌍 Real World

In real Airflow setups, secrets like passwords and API keys are stored securely and accessed at runtime to avoid exposing sensitive data in code.

💼 Career

Understanding secrets management is crucial for DevOps roles to maintain security and compliance when deploying workflows and applications.

1

Create a dictionary to simulate Airflow Variables

Create a dictionary called airflow_variables with these exact entries: 'db_password': 'SuperSecret123', 'api_key': 'XYZ-987654'

Apache Airflow

# Create the airflow_variables dictionary with secrets
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with the exact keys and values.

2

Add a configuration variable for the secret key

Create a variable called secret_key and set it to the string 'db_password'

Apache Airflow

airflow_variables = {'db_password': 'SuperSecret123', 'api_key': 'XYZ-987654'}
# Create the secret_key variable with the value 'db_password'
# Your code here

Need a hint?

Assign the string 'db_password' to the variable secret_key.

3

Write a function to retrieve the secret

Define a function called get_secret that takes a parameter key and returns the value from airflow_variables for that key

Apache Airflow

airflow_variables = {'db_password': 'SuperSecret123', 'api_key': 'XYZ-987654'}
secret_key = 'db_password'
# Define get_secret(key) to return airflow_variables[key]
# Your code here

Need a hint?

Use def to define the function and return the dictionary value for the given key.

4

Print the retrieved secret value

Write a print statement that calls get_secret(secret_key) and prints the result

Apache Airflow

airflow_variables = {'db_password': 'SuperSecret123', 'api_key': 'XYZ-987654'}
secret_key = 'db_password'

def get_secret(key):
    return airflow_variables[key]

# Print the secret value using get_secret(secret_key)
# Your code here

Need a hint?

Use print(get_secret(secret_key)) to display the secret.