Connection management for cloud services in Apache Airflow - Commands & Configuration

Introduction

Managing connections in Airflow helps you securely store and reuse credentials and settings needed to access cloud services. This avoids hardcoding sensitive information in your workflows and makes your automation safer and easier to maintain.

When you need to connect Airflow tasks to AWS services like S3 or Redshift without exposing keys in code

When you want to reuse the same cloud credentials across multiple workflows or DAGs

When you want to update cloud service credentials in one place without changing every workflow

When you want to keep your cloud connection details encrypted and secure within Airflow

When you want to switch between different cloud accounts or environments easily

Commands

This command adds a new Airflow connection named 'my_aws_conn' for AWS. It stores your AWS access key and secret key securely, along with the region information, so your tasks can use it to access AWS services.

Terminal

airflow connections add my_aws_conn --conn-type aws --conn-login AWS_ACCESS_KEY_ID --conn-password AWS_SECRET_ACCESS_KEY --conn-extra '{"region_name": "us-east-1"}'

Expected Output

Added connection `my_aws_conn`

→

--conn-type - Specifies the type of connection, here 'aws' for Amazon Web Services

→

--conn-login - Sets the username or access key for the connection

→

--conn-password - Sets the password or secret key for the connection

→

--conn-extra - Adds extra JSON-formatted parameters like region

This command retrieves and shows the details of the 'my_aws_conn' connection to verify it was added correctly.

Terminal

airflow connections get my_aws_conn

Expected Output

Conn Id: my_aws_conn

Conn Type: aws

Host: None

Login: AWS_ACCESS_KEY_ID

Password: AWS_SECRET_ACCESS_KEY

Schema: None

Port: None

Extra: {"region_name": "us-east-1"}

This command runs a task from your Airflow DAG that uses the 'my_aws_conn' connection to access AWS services during execution.

Terminal

airflow tasks run example_dag example_task 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {taskinstance.py:1234} INFO - Executing task example_task using connection my_aws_conn

[2024-06-01 12:00:05,000] {taskinstance.py:5678} INFO - Task completed successfully

This command deletes the 'my_aws_conn' connection when it is no longer needed, cleaning up your Airflow environment.

Terminal

airflow connections delete my_aws_conn

Expected Output

Deleted connection `my_aws_conn`

Key Concept

If you remember nothing else from connection management, remember: store your cloud credentials securely in Airflow connections to reuse them safely across workflows.

Common Mistakes

Hardcoding cloud credentials directly in DAG code

This exposes sensitive information and makes updating credentials difficult and risky

Use Airflow connections to store credentials securely and reference them in your DAGs

Using incorrect connection IDs or typos when referencing connections in tasks

Tasks will fail because they cannot find the connection details

Double-check connection IDs and use airflow connections get to verify them

Not setting required extra parameters like region in the connection

Cloud service clients may fail to connect or use wrong defaults

Include all necessary extra JSON parameters when creating connections

Summary

Use 'airflow connections add' to securely store cloud service credentials and settings.

Verify connections with 'airflow connections get' before running workflows.

Reference these connections in your DAG tasks to avoid hardcoding sensitive data.

Clean up unused connections with 'airflow connections delete' to keep your environment tidy.

Practice

1. What is the main purpose of managing connections in Airflow for cloud services?

easy

A. To manually enter credentials every time a task runs

B. To write custom code for each cloud service login

C. To store cloud login details securely and reuse them in tasks

D. To disable cloud service access in Airflow

Solution

1. Step 1: Understand connection management role

   Connection management stores credentials securely so tasks can access cloud services without exposing secrets.

2. Step 2: Identify reuse benefit

   Reusing connections avoids repeating credentials in multiple tasks, making workflows cleaner and safer.

3. Final Answer:

   To store cloud login details securely and reuse them in tasks -> Option C

4. Quick Check:

   Secure storage + reuse = A [OK]

Hint: Connections store credentials once for many tasks [OK]

Common Mistakes:

• Thinking credentials must be typed in each task
• Assuming connections are only for UI use
• Believing connections disable cloud access

2. Which command correctly adds a Google Cloud connection named 'gcp_default' in Airflow CLI?

easy

A. airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}'

B. airflow add connection gcp_default --type gcp --extra '{"key_path":"/path/key.json"}'

C. airflow connections create gcp_default --type google --extra '{"key":"/path/key.json"}'

D. airflow connection add gcp_default --conn-type gcp --conn-extra '{"key_path":"/path/key.json"}'

Solution

1. Step 1: Recall correct Airflow CLI syntax

   The command to add connections is 'airflow connections add' followed by connection id and parameters.

2. Step 2: Match correct flags and JSON format

   airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}' uses '--conn-type google_cloud_platform' and properly formatted JSON for extra parameters, matching Airflow docs.

3. Final Answer:

   airflow connections add 'gcp_default' --conn-type google_cloud_platform --conn-extra '{"extra__google_cloud_platform__key_path":"/path/key.json"}' -> Option A

4. Quick Check:

   Correct command syntax = C [OK]

Hint: Use 'airflow connections add' with correct flags [OK]

Common Mistakes:

• Using 'airflow add connection' instead of 'connections add'
• Wrong JSON key names in --conn-extra
• Missing quotes around JSON string

3. Given this Airflow DAG snippet using a connection ID 'aws_conn', what will happen if the connection is missing?

from airflow.providers.amazon.aws.operators.s3 import S3ListOperator

list_s3 = S3ListOperator(
    task_id='list_s3',
    bucket='my-bucket',
    aws_conn_id='aws_conn'
)

medium

A. The task will create a new connection automatically

B. The task will fail with a connection not found error

C. The task will skip without error

D. The task will run using default AWS credentials

Solution

1. Step 1: Understand connection usage in operator

   The operator uses 'aws_conn_id' to find credentials. If missing, it cannot authenticate.

2. Step 2: Predict behavior on missing connection

   Airflow raises an error when a required connection is not found, causing task failure.

3. Final Answer:

   The task will fail with a connection not found error -> Option B

4. Quick Check:

   Missing connection causes failure = D [OK]

Hint: Missing connection ID causes task failure [OK]

Common Mistakes:

• Assuming default credentials are used automatically
• Thinking task skips if connection missing
• Believing Airflow auto-creates connections

4. You added a connection with the command:

airflow connections add 'azure_default' --conn-type azure --conn-extra '{"tenantId":"abc","clientId":"123"}'

But your tasks fail to authenticate. What is the likely mistake?

medium

A. The JSON keys in --conn-extra are missing the 'extra__azure__' prefix

B. The connection type should be 'azure_cosmos' not 'azure'

C. The connection ID should not be quoted

D. The command should use 'airflow connection add' instead of 'connections add'

Solution

1. Step 1: Check connection type and extra format

   Azure connections require extra keys prefixed with 'extra__azure__' to be recognized properly.

2. Step 2: Identify JSON key format error

   Missing prefix in JSON keys causes Airflow to ignore credentials, leading to authentication failure.

3. Final Answer:

   The JSON keys in --conn-extra are missing the 'extra__azure__' prefix -> Option A

4. Quick Check:

   Correct extra key format fixes auth = B [OK]

Hint: Use 'extra__service__' prefix in --conn-extra keys [OK]

Common Mistakes:

• Using wrong connection type name
• Not prefixing JSON keys correctly
• Wrong CLI command syntax

5. You want to securely manage multiple cloud connections (AWS, GCP, Azure) in Airflow and reuse them in different DAGs. Which approach is best?

hard

A. Use environment variables only and avoid Airflow connections

B. Hardcode credentials in each DAG file for each cloud service

C. Store all credentials in a single connection with mixed cloud keys

D. Create separate connections for each cloud service using Airflow UI or CLI, then reference their IDs in tasks

Solution

1. Step 1: Understand best practice for connection management

   Separate connections per cloud service keep credentials organized and secure.

2. Step 2: Recognize reuse and security benefits

   Referencing connection IDs in tasks avoids duplication and keeps secrets out of code.

3. Final Answer:

   Create separate connections for each cloud service using Airflow UI or CLI, then reference their IDs in tasks -> Option D

4. Quick Check:

   Separate connections + reuse = A [OK]

Hint: Use separate connections per cloud, reference by ID [OK]

Common Mistakes:

• Hardcoding credentials in DAGs
• Mixing all credentials in one connection
• Avoiding Airflow connections for secrets