Secrets management in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary purpose of Airflow's secrets backend?

Airflow supports multiple secrets backends like HashiCorp Vault, AWS Secrets Manager, and environment variables. What is the main reason to use a secrets backend in Airflow?

A To automatically generate logs for task failures.

B To speed up DAG parsing by caching task metadata.

C To securely store and retrieve sensitive information like passwords and API keys during task execution.

D To schedule DAG runs based on secret expiration dates.

💡 Hint

Think about why you wouldn't want to hardcode passwords in your DAG files.

Secrets backends in Airflow are designed to keep sensitive data safe and provide it only when needed during task runs.

💻 Command Output

intermediate

What is the output of this Airflow CLI command?

Given that Airflow is configured with the environment variables secrets backend, what will this command output?

Apache Airflow

airflow secrets get --conn-id my_postgres_conn

A Empty output because secrets backends do not support CLI retrieval

B A JSON object with all connection details printed to the console

C SyntaxError: invalid command usage

D Connection URI string stored in the environment variable AIRFLOW_CONN_MY_POSTGRES_CONN

💡 Hint

Check how Airflow environment variable secrets are named and accessed.

The CLI command retrieves the connection URI from the environment variable named AIRFLOW_CONN_MY_POSTGRES_CONN when using the environment variables secrets backend.

❓ Configuration

advanced

Which configuration snippet correctly enables HashiCorp Vault as Airflow's secrets backend?

Choose the correct airflow.cfg snippet to configure HashiCorp Vault as the secrets backend with the Vault URL at https://vault.example.com and token authentication.

A

[secrets]
backend = airflow.providers.hashicorp.vault.VaultBackend
backend_kwargs = {"url": "vault.example.com", "auth": "token"}

B

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"url": "https://vault.example.com", "token": "mytoken"}

C

[secrets]
backend = airflow.secrets.vault.VaultBackend
backend_kwargs = {"url": "https://vault.example.com", "token": "mytoken"}

D

[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {"endpoint": "https://vault.example.com", "token": "mytoken"}

💡 Hint

Check the exact import path and parameter names for VaultBackend in Airflow providers.

The correct backend path is airflow.providers.hashicorp.secrets.vault.VaultBackend and the backend_kwargs keys must match the expected parameters like 'url' and 'token'.

❓ Troubleshoot

advanced

Why does Airflow fail to retrieve secrets from AWS Secrets Manager?

An Airflow deployment uses AWS Secrets Manager as the secrets backend. The DAGs fail with errors saying secrets are missing. Which of these is the most likely cause?

A The IAM role or user running Airflow lacks permissions to read secrets from AWS Secrets Manager.

B The secrets backend is not supported by Airflow and must be disabled.

C The secrets are stored in AWS S3 instead of AWS Secrets Manager.

D Airflow requires secrets to be stored only in environment variables.

💡 Hint

Think about AWS permissions needed for Airflow to access secrets.

Airflow needs proper AWS IAM permissions to read secrets from AWS Secrets Manager. Without them, it cannot retrieve secrets and will fail.

🔀 Workflow

expert

What is the correct order to securely use secrets in an Airflow DAG?

Arrange these steps in the correct order to securely use a database password stored in a secrets backend within an Airflow DAG.

A 1,2,3,4

B 2,1,3,4

C 1,3,2,4

D 3,1,2,4

💡 Hint

Think about what must exist before Airflow can retrieve secrets and how tasks use them.

First, the secret must be stored securely. Then Airflow must be configured to access it. Next, DAGs reference the secret, and finally tasks use it without hardcoding.

Practice

1. What is the main purpose of secrets management in Apache Airflow?

easy

A. To keep sensitive information like passwords out of the code

B. To speed up DAG execution

C. To store large datasets

D. To monitor task progress visually

Solution

1. Step 1: Understand what secrets management means

   Secrets management is about protecting sensitive data such as passwords or API keys from being exposed in code.

2. Step 2: Identify the main goal in Airflow context

   Airflow uses secrets management to keep such sensitive info safe and separate from DAG code.

3. Final Answer:

   To keep sensitive information like passwords out of the code -> Option A

4. Quick Check:

   Secrets management = protect sensitive info [OK]

Hint: Secrets management means hiding passwords from code [OK]

Common Mistakes:

• Confusing secrets management with data storage
• Thinking it speeds up DAGs
• Assuming it monitors tasks

2. Which of the following is the correct way to retrieve a secret stored as an Airflow Variable in a DAG?

easy

A. get_variable('my_secret')

B. Variable.get('my_secret')

C. Variable.fetch('my_secret')

D. fetch_variable('my_secret')

Solution

1. Step 1: Recall Airflow's Variable API

   Airflow provides a Variable class with a get() method to access stored variables or secrets.

2. Step 2: Match the correct syntax

   The correct method call is Variable.get('variable_name'), so Variable.get('my_secret') is correct.

3. Final Answer:

   Variable.get('my_secret') -> Option B

4. Quick Check:

   Use Variable.get() to access secrets [OK]

Hint: Use Variable.get() to fetch secrets in Airflow [OK]

Common Mistakes:

• Using non-existent methods like fetch or get_variable
• Confusing function names
• Missing the Variable class prefix

3. Given this Airflow DAG snippet, what will be the output if the secret 'db_password' is stored as 'mypassword123' in Airflow Variables?

from airflow.models import Variable

password = Variable.get('db_password')
print(password)

medium

A. Error: Variable not found

B. db_password

C. Variable.get('db_password')

D. mypassword123

Solution

1. Step 1: Understand Variable.get() behavior

   Variable.get('db_password') fetches the value stored under 'db_password' key, which is 'mypassword123'.

2. Step 2: Predict the print output

   Printing the variable will output the string 'mypassword123'.

3. Final Answer:

   mypassword123 -> Option D

4. Quick Check:

   Variable.get() returns stored secret value [OK]

Hint: Variable.get() returns the secret's stored value [OK]

Common Mistakes:

• Thinking it prints the key name
• Expecting the method call as output
• Assuming an error without checking storage

4. You wrote this code in your Airflow DAG to get a secret:

from airflow.models import Variable

api_key = Variable.get('api_key')

But you get an error: "KeyError: 'api_key' not found". What is the best way to fix this?

medium

A. Add a default value like Variable.get('api_key', default_var='default')

B. Change Variable.get to Variable.fetch

C. Remove the import statement

D. Rename 'api_key' to 'API_KEY'

Solution

1. Step 1: Understand the error cause

   The error means 'api_key' variable does not exist in Airflow Variables, so Variable.get() fails.

2. Step 2: Use default_var to avoid error

   Adding default_var='default' returns 'default' if the variable is missing, preventing the error.

3. Final Answer:

   Add a default value like Variable.get('api_key', default_var='default') -> Option A

4. Quick Check:

   Use default_var to handle missing secrets safely [OK]

Hint: Use default_var to avoid missing variable errors [OK]

Common Mistakes:

• Using non-existent Variable.fetch method
• Removing imports breaks code
• Changing variable name without updating storage

5. You want to securely store and access AWS credentials in Airflow using AWS Secrets Manager. Which approach correctly integrates this with Airflow's secrets backend?

hard

A. Hardcode AWS credentials in DAG files for faster access

B. Store credentials in Airflow Variables and access with Variable.get() only

C. Configure airflow.cfg with [secrets] backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

D. Use environment variables without any Airflow configuration

Solution

1. Step 1: Identify how Airflow integrates with AWS Secrets Manager

   Airflow supports multiple secrets backends configured in airflow.cfg under the [secrets] section.

2. Step 2: Correctly configure the secrets backend

   Setting backend to airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend enables Airflow to fetch secrets from AWS Secrets Manager securely.

3. Final Answer:

   Configure airflow.cfg with [secrets] backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend -> Option C

4. Quick Check:

   Configure secrets backend for AWS Secrets Manager [OK]

Hint: Set secrets backend in airflow.cfg for AWS Secrets Manager [OK]

Common Mistakes:

• Hardcoding credentials in DAGs (unsafe)
• Using Variables only without backend config
• Ignoring Airflow secrets backend configuration