Secrets management in Apache Airflow - Step-by-Step Execution

Process Flow - Secrets management

Define Secret

→Store Secret Securely

↓

Configure Airflow to Access Secrets

↓

Airflow Task Requests Secret

↓

Secret Retrieved from Backend

↓

Task Uses Secret Securely

↓

Secret Not Logged or Exposed

↓

End

This flow shows how secrets are defined, stored securely, accessed by Airflow tasks, and used without exposure.

Execution Sample

Apache Airflow

from airflow.models import Variable
secret = Variable.get('db_password', deserialize_json=False)
print(f"Using secret: {secret}")

This code fetches a secret named 'db_password' from Airflow's secret backend and prints it.

Process Table

Step	Action	Airflow Variable.get Call	Secret Retrieved	Output/Effect
1	Call Variable.get('db_password')	Variable.get('db_password')	retrieves 's3cr3tP@ss'	Secret stored in variable 'secret'
2	Print secret value	print(f"Using secret: {secret}")	N/A	Output: Using secret: s3cr3tP@ss
3	End of task	N/A	N/A	Secret used securely, not logged by Airflow internally

💡 Task completes after secret is retrieved and used without exposure.

Status Tracker

Variable	Start	After Step 1	After Step 2	Final
secret	undefined	s3cr3tP@ss	s3cr3tP@ss	s3cr3tP@ss

Key Moments - 3 Insights

Why doesn't Airflow log the secret value when Variable.get is called?

Airflow's Variable.get method retrieves the secret but does not log its value to protect sensitive data, as shown in execution_table step 1 and 3.

What happens if the secret 'db_password' is not found?

Variable.get would raise an error or return None if configured, stopping the task or requiring a default; this is not shown here but is important for error handling.

How does Airflow keep secrets secure when tasks run?

Secrets are fetched at runtime from secure backends and stored only in memory variables like 'secret', avoiding writing to logs or files, as seen in variable_tracker and execution_table.

Visual Quiz - 3 Questions

Look at the execution table, what value does the variable 'secret' hold after step 1?

A s3cr3tP@ss

B undefined

C None

D Variable name 'db_password'

💡 Hint

Check the 'Secret Retrieved' column in row for step 1 in execution_table.

At which step does the secret get printed to output?

A Step 1

B Step 2

C Step 3

D Secret is never printed

💡 Hint

Look at the 'Output/Effect' column in execution_table for step 2.

If the secret was not found, what would likely happen to the variable 'secret'?

A It would hold an empty string

B It would hold the string 's3cr3tP@ss'

C It would be undefined or cause an error

D It would automatically generate a new secret

💡 Hint

Refer to key_moments about missing secrets and error handling.

Concept Snapshot

Secrets management in Airflow:
- Store secrets securely outside DAG code
- Use Variable.get('secret_name') to fetch secrets
- Secrets are retrieved at runtime, not logged
- Handle missing secrets to avoid task failure
- Keep secrets in memory only during task execution

Full Transcript

This visual execution shows how Airflow manages secrets. First, a secret named 'db_password' is defined and stored securely. Airflow is configured to access secrets from its backend. When a task runs, it calls Variable.get('db_password') to retrieve the secret value 's3cr3tP@ss'. The secret is stored in a variable named 'secret' in memory. The task then prints the secret value to output. Airflow does not log the secret internally to protect it. The secret is used securely and the task ends. Key points include that secrets are fetched at runtime, not exposed in logs, and missing secrets can cause errors. This ensures sensitive data like passwords stay safe during workflow execution.

Practice

1. What is the main purpose of secrets management in Apache Airflow?

easy

A. To keep sensitive information like passwords out of the code

B. To speed up DAG execution

C. To store large datasets

D. To monitor task progress visually

Solution

1. Step 1: Understand what secrets management means

   Secrets management is about protecting sensitive data such as passwords or API keys from being exposed in code.

2. Step 2: Identify the main goal in Airflow context

   Airflow uses secrets management to keep such sensitive info safe and separate from DAG code.

3. Final Answer:

   To keep sensitive information like passwords out of the code -> Option A

4. Quick Check:

   Secrets management = protect sensitive info [OK]

Hint: Secrets management means hiding passwords from code [OK]

Common Mistakes:

• Confusing secrets management with data storage
• Thinking it speeds up DAGs
• Assuming it monitors tasks

2. Which of the following is the correct way to retrieve a secret stored as an Airflow Variable in a DAG?

easy

A. get_variable('my_secret')

B. Variable.get('my_secret')

C. Variable.fetch('my_secret')

D. fetch_variable('my_secret')

Solution

1. Step 1: Recall Airflow's Variable API

   Airflow provides a Variable class with a get() method to access stored variables or secrets.

2. Step 2: Match the correct syntax

   The correct method call is Variable.get('variable_name'), so Variable.get('my_secret') is correct.

3. Final Answer:

   Variable.get('my_secret') -> Option B

4. Quick Check:

   Use Variable.get() to access secrets [OK]

Hint: Use Variable.get() to fetch secrets in Airflow [OK]

Common Mistakes:

• Using non-existent methods like fetch or get_variable
• Confusing function names
• Missing the Variable class prefix

3. Given this Airflow DAG snippet, what will be the output if the secret 'db_password' is stored as 'mypassword123' in Airflow Variables?

from airflow.models import Variable

password = Variable.get('db_password')
print(password)

medium

A. Error: Variable not found

B. db_password

C. Variable.get('db_password')

D. mypassword123

Solution

1. Step 1: Understand Variable.get() behavior

   Variable.get('db_password') fetches the value stored under 'db_password' key, which is 'mypassword123'.

2. Step 2: Predict the print output

   Printing the variable will output the string 'mypassword123'.

3. Final Answer:

   mypassword123 -> Option D

4. Quick Check:

   Variable.get() returns stored secret value [OK]

Hint: Variable.get() returns the secret's stored value [OK]

Common Mistakes:

• Thinking it prints the key name
• Expecting the method call as output
• Assuming an error without checking storage

4. You wrote this code in your Airflow DAG to get a secret:

from airflow.models import Variable

api_key = Variable.get('api_key')

But you get an error: "KeyError: 'api_key' not found". What is the best way to fix this?

medium

A. Add a default value like Variable.get('api_key', default_var='default')

B. Change Variable.get to Variable.fetch

C. Remove the import statement

D. Rename 'api_key' to 'API_KEY'

Solution

1. Step 1: Understand the error cause

   The error means 'api_key' variable does not exist in Airflow Variables, so Variable.get() fails.

2. Step 2: Use default_var to avoid error

   Adding default_var='default' returns 'default' if the variable is missing, preventing the error.

3. Final Answer:

   Add a default value like Variable.get('api_key', default_var='default') -> Option A

4. Quick Check:

   Use default_var to handle missing secrets safely [OK]

Hint: Use default_var to avoid missing variable errors [OK]

Common Mistakes:

• Using non-existent Variable.fetch method
• Removing imports breaks code
• Changing variable name without updating storage

5. You want to securely store and access AWS credentials in Airflow using AWS Secrets Manager. Which approach correctly integrates this with Airflow's secrets backend?

hard

A. Hardcode AWS credentials in DAG files for faster access

B. Store credentials in Airflow Variables and access with Variable.get() only

C. Configure airflow.cfg with [secrets] backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

D. Use environment variables without any Airflow configuration

Solution

1. Step 1: Identify how Airflow integrates with AWS Secrets Manager

   Airflow supports multiple secrets backends configured in airflow.cfg under the [secrets] section.

2. Step 2: Correctly configure the secrets backend

   Setting backend to airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend enables Airflow to fetch secrets from AWS Secrets Manager securely.

3. Final Answer:

   Configure airflow.cfg with [secrets] backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend -> Option C

4. Quick Check:

   Configure secrets backend for AWS Secrets Manager [OK]

Hint: Set secrets backend in airflow.cfg for AWS Secrets Manager [OK]

Common Mistakes:

• Hardcoding credentials in DAGs (unsafe)
• Using Variables only without backend config
• Ignoring Airflow secrets backend configuration