Connection management for cloud services in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Airflow Connections for Cloud Services

What is the primary purpose of using Airflow Connections when working with cloud services?

A To write Python code that interacts with cloud APIs directly inside DAGs.

B To schedule DAG runs at specific times automatically.

C To securely store and manage credentials and connection details for cloud services.

D To monitor the health of cloud services in real-time.

💡 Hint

Think about how Airflow keeps sensitive information safe and reusable.

Airflow Connections provide a centralized and secure way to store credentials and connection info, so DAGs can use them without hardcoding sensitive data.

💻 Command Output

intermediate

Output of Listing Airflow Connections

What is the output of the following command if you have one connection named 'aws_default' configured?

airflow connections list

A A table listing connection IDs including 'aws_default' with its type and extra info.

B An error saying 'command not found'.

C A blank output with no connections listed.

D A JSON string with connection details.

💡 Hint

Think about what the 'list' command shows in Airflow CLI.

The 'airflow connections list' command outputs a table of all configured connections, including their IDs and types.

❓ Configuration

advanced

Correct Airflow Connection URI for Google Cloud Storage

Which of the following is the correct Airflow connection URI format to connect to Google Cloud Storage using a service account JSON key file?

A google-cloud-platform://?extra__google_cloud_platform__key_path=/path/to/key.json

B gcp://service_account.json

C gs://service_account.json

D google://keyfile=/path/to/key.json

💡 Hint

Look for the correct prefix and how extra parameters are passed in Airflow connection URIs.

Airflow uses 'google-cloud-platform' as the connection type and passes the key file path in the 'extra' field with the prefix 'extra__google_cloud_platform__key_path'.

❓ Troubleshoot

advanced

Troubleshooting Airflow Connection Authentication Failure

You have set up an Airflow connection to AWS but your DAG fails with an authentication error. Which of the following is the most likely cause?

A The AWS region is not specified in the DAG code.

B The AWS access key and secret key are missing or incorrect in the Airflow connection configuration.

C The Airflow scheduler is not running.

D The DAG file is missing the import statement for the AWS hook.

💡 Hint

Authentication errors usually relate to credentials.

If the AWS keys are missing or wrong in the connection, Airflow cannot authenticate to AWS services, causing failures.

🔀 Workflow

expert

Best Practice Workflow for Managing Cloud Service Connections in Airflow

Which workflow correctly describes the best practice for managing cloud service connections securely and efficiently in Airflow?

A Use public cloud credentials without restrictions to simplify connection setup.

B Hardcode credentials in DAG files for easy access and version control.

C Store credentials in plain text files on the Airflow server and read them in DAGs.

D Store credentials in Airflow Connections, use environment variables for sensitive info, and restrict access to Airflow UI and metadata DB.

💡 Hint

Think about security and maintainability in managing credentials.

Best practice is to store credentials securely in Airflow Connections, use environment variables for sensitive data, and limit access to prevent leaks.

Practice

1. What is the main purpose of managing connections in Airflow for cloud services?

easy

A. To manually enter credentials every time a task runs

B. To write custom code for each cloud service login

C. To store cloud login details securely and reuse them in tasks

D. To disable cloud service access in Airflow

Solution

1. Step 1: Understand connection management role

   Connection management stores credentials securely so tasks can access cloud services without exposing secrets.

2. Step 2: Identify reuse benefit

   Reusing connections avoids repeating credentials in multiple tasks, making workflows cleaner and safer.

3. Final Answer:

   To store cloud login details securely and reuse them in tasks -> Option C

4. Quick Check:

   Secure storage + reuse = A [OK]

Hint: Connections store credentials once for many tasks [OK]

Common Mistakes:

• Thinking credentials must be typed in each task
• Assuming connections are only for UI use
• Believing connections disable cloud access

2. Which command correctly adds a Google Cloud connection named 'gcp_default' in Airflow CLI?

easy

A. airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}'

B. airflow add connection gcp_default --type gcp --extra '{"key_path":"/path/key.json"}'

C. airflow connections create gcp_default --type google --extra '{"key":"/path/key.json"}'

D. airflow connection add gcp_default --conn-type gcp --conn-extra '{"key_path":"/path/key.json"}'

Solution

1. Step 1: Recall correct Airflow CLI syntax

   The command to add connections is 'airflow connections add' followed by connection id and parameters.

2. Step 2: Match correct flags and JSON format

   airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}' uses '--conn-type google_cloud_platform' and properly formatted JSON for extra parameters, matching Airflow docs.

3. Final Answer:

   airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}' -> Option A

4. Quick Check:

   Correct command syntax = C [OK]

Hint: Use 'airflow connections add' with correct flags [OK]

Common Mistakes:

• Using 'airflow add connection' instead of 'connections add'
• Wrong JSON key names in --conn-extra
• Missing quotes around JSON string

3. Given this Airflow DAG snippet using a connection ID 'aws_conn', what will happen if the connection is missing?

from airflow.providers.amazon.aws.operators.s3 import S3ListOperator

list_s3 = S3ListOperator(
    task_id='list_s3',
    bucket='my-bucket',
    aws_conn_id='aws_conn'
)

medium

A. The task will create a new connection automatically

B. The task will fail with a connection not found error

C. The task will skip without error

D. The task will run using default AWS credentials

Solution

1. Step 1: Understand connection usage in operator

   The operator uses 'aws_conn_id' to find credentials. If missing, it cannot authenticate.

2. Step 2: Predict behavior on missing connection

   Airflow raises an error when a required connection is not found, causing task failure.

3. Final Answer:

   The task will fail with a connection not found error -> Option B

4. Quick Check:

   Missing connection causes failure = D [OK]

Hint: Missing connection ID causes task failure [OK]

Common Mistakes:

• Assuming default credentials are used automatically
• Thinking task skips if connection missing
• Believing Airflow auto-creates connections

4. You added a connection with the command:

airflow connections add 'azure_default' --conn-type azure --conn-extra '{"tenantId":"abc","clientId":"123"}'

But your tasks fail to authenticate. What is the likely mistake?

medium

A. The JSON keys in --conn-extra are missing the 'extra__azure__' prefix

B. The connection type should be 'azure_cosmos' not 'azure'

C. The connection ID should not be quoted

D. The command should use 'airflow connection add' instead of 'connections add'

Solution

1. Step 1: Check connection type and extra format

   Azure connections require extra keys prefixed with 'extra__azure__' to be recognized properly.

2. Step 2: Identify JSON key format error

   Missing prefix in JSON keys causes Airflow to ignore credentials, leading to authentication failure.

3. Final Answer:

   The JSON keys in --conn-extra are missing the 'extra__azure__' prefix -> Option A

4. Quick Check:

   Correct extra key format fixes auth = B [OK]

Hint: Use 'extra__service__' prefix in --conn-extra keys [OK]

Common Mistakes:

• Using wrong connection type name
• Not prefixing JSON keys correctly
• Wrong CLI command syntax

5. You want to securely manage multiple cloud connections (AWS, GCP, Azure) in Airflow and reuse them in different DAGs. Which approach is best?

hard

A. Use environment variables only and avoid Airflow connections

B. Hardcode credentials in each DAG file for each cloud service

C. Store all credentials in a single connection with mixed cloud keys

D. Create separate connections for each cloud service using Airflow UI or CLI, then reference their IDs in tasks

Solution

1. Step 1: Understand best practice for connection management

   Separate connections per cloud service keep credentials organized and secure.

2. Step 2: Recognize reuse and security benefits

   Referencing connection IDs in tasks avoids duplication and keeps secrets out of code.

3. Final Answer:

   Create separate connections for each cloud service using Airflow UI or CLI, then reference their IDs in tasks -> Option D

4. Quick Check:

   Separate connections + reuse = A [OK]

Hint: Use separate connections per cloud, reference by ID [OK]

Common Mistakes:

• Hardcoding credentials in DAGs
• Mixing all credentials in one connection
• Avoiding Airflow connections for secrets