Secrets management in Apache Airflow - Step-by-Step Execution

Process Flow - Secrets management

Define Secret

→Store Secret Securely

↓

Configure Airflow to Access Secrets

↓

Airflow Task Requests Secret

↓

Secret Retrieved from Backend

↓

Task Uses Secret Securely

↓

Secret Not Logged or Exposed

↓

End

This flow shows how secrets are defined, stored securely, accessed by Airflow tasks, and used without exposure.

Execution Sample

Apache Airflow

from airflow.models import Variable
secret = Variable.get('db_password', deserialize_json=False)
print(f"Using secret: {secret}")

This code fetches a secret named 'db_password' from Airflow's secret backend and prints it.

Process Table

Step	Action	Airflow Variable.get Call	Secret Retrieved	Output/Effect
1	Call Variable.get('db_password')	Variable.get('db_password')	retrieves 's3cr3tP@ss'	Secret stored in variable 'secret'
2	Print secret value	print(f"Using secret: {secret}")	N/A	Output: Using secret: s3cr3tP@ss
3	End of task	N/A	N/A	Secret used securely, not logged by Airflow internally

💡 Task completes after secret is retrieved and used without exposure.

Status Tracker

Variable	Start	After Step 1	After Step 2	Final
secret	undefined	s3cr3tP@ss	s3cr3tP@ss	s3cr3tP@ss

Key Moments - 3 Insights

Why doesn't Airflow log the secret value when Variable.get is called?

Airflow's Variable.get method retrieves the secret but does not log its value to protect sensitive data, as shown in execution_table step 1 and 3.

What happens if the secret 'db_password' is not found?

Variable.get would raise an error or return None if configured, stopping the task or requiring a default; this is not shown here but is important for error handling.

How does Airflow keep secrets secure when tasks run?

Secrets are fetched at runtime from secure backends and stored only in memory variables like 'secret', avoiding writing to logs or files, as seen in variable_tracker and execution_table.

Visual Quiz - 3 Questions

Look at the execution table, what value does the variable 'secret' hold after step 1?

A s3cr3tP@ss

B undefined

C None

D Variable name 'db_password'

💡 Hint

Check the 'Secret Retrieved' column in row for step 1 in execution_table.

At which step does the secret get printed to output?

A Step 1

B Step 2

C Step 3

D Secret is never printed

💡 Hint

Look at the 'Output/Effect' column in execution_table for step 2.

If the secret was not found, what would likely happen to the variable 'secret'?

A It would hold an empty string

B It would hold the string 's3cr3tP@ss'

C It would be undefined or cause an error

D It would automatically generate a new secret

💡 Hint

Refer to key_moments about missing secrets and error handling.

Concept Snapshot

Secrets management in Airflow:
- Store secrets securely outside DAG code
- Use Variable.get('secret_name') to fetch secrets
- Secrets are retrieved at runtime, not logged
- Handle missing secrets to avoid task failure
- Keep secrets in memory only during task execution

Full Transcript

This visual execution shows how Airflow manages secrets. First, a secret named 'db_password' is defined and stored securely. Airflow is configured to access secrets from its backend. When a task runs, it calls Variable.get('db_password') to retrieve the secret value 's3cr3tP@ss'. The secret is stored in a variable named 'secret' in memory. The task then prints the secret value to output. Airflow does not log the secret internally to protect it. The secret is used securely and the task ends. Key points include that secrets are fetched at runtime, not exposed in logs, and missing secrets can cause errors. This ensures sensitive data like passwords stay safe during workflow execution.