Connection encryption in Apache Airflow - Mini Project: Build & Apply

Connection encryption in Apache Airflow

📖 Scenario: You are setting up Apache Airflow to securely connect to a PostgreSQL database. To protect sensitive data, you need to configure connection encryption.

🎯 Goal: Configure an Airflow connection with encryption enabled to securely connect to a PostgreSQL database.

📋 What You'll Learn

Create an Airflow connection dictionary with exact parameters

Add an encryption flag to the connection configuration

Use the Airflow CLI command to add the connection with encryption

Print the connection details to verify encryption is enabled

💡 Why This Matters

🌍 Real World

In real projects, encrypting connections protects sensitive data when Airflow talks to databases or services.

💼 Career

Knowing how to configure encrypted connections is essential for DevOps roles managing secure data pipelines.

1

Create Airflow connection dictionary

Create a dictionary called conn_postgres with these exact keys and values: conn_id set to "my_postgres_conn", conn_type set to "postgres", host set to "localhost", login set to "airflow_user", password set to "airflow_pass", and schema set to "airflow_db".

Apache Airflow

# Create the Airflow connection dictionary with exact keys and values
# Your code here

Hint

Use a Python dictionary with the exact keys and values as specified.

2

Add encryption flag to connection dictionary

Add a key extra to the conn_postgres dictionary with the exact value '{"sslmode": "require"}' to enable encryption.

Apache Airflow

conn_postgres = {
    "conn_id": "my_postgres_conn",
    "conn_type": "postgres",
    "host": "localhost",
    "login": "airflow_user",
    "password": "airflow_pass",
    "schema": "airflow_db"
}
# Add the 'extra' key with encryption settings below
# Your code here

Hint

The extra key must be a string containing JSON with "sslmode": "require".

3

Add the connection using Airflow CLI command

Write a string variable called cli_command that contains the exact Airflow CLI command to add the connection with encryption: airflow connections add my_postgres_conn --conn-type postgres --conn-host localhost --conn-login airflow_user --conn-password airflow_pass --conn-schema airflow_db --conn-extra '{"sslmode": "require"}'

Apache Airflow

conn_postgres = {
    "conn_id": "my_postgres_conn",
    "conn_type": "postgres",
    "host": "localhost",
    "login": "airflow_user",
    "password": "airflow_pass",
    "schema": "airflow_db",
    "extra": '{"sslmode": "require"}'
}
# Write the Airflow CLI command string to add the connection below
# Your code here

Hint

Escape double quotes inside the conn-extra JSON string properly.

4

Print the connection dictionary to verify encryption

Write a print statement to display the conn_postgres dictionary.

Apache Airflow

conn_postgres = {
    "conn_id": "my_postgres_conn",
    "conn_type": "postgres",
    "host": "localhost",
    "login": "airflow_user",
    "password": "airflow_pass",
    "schema": "airflow_db",
    "extra": '{"sslmode": "require"}'
}

cli_command = "airflow connections add my_postgres_conn --conn-type postgres --conn-host localhost --conn-login airflow_user --conn-password airflow_pass --conn-schema airflow_db --conn-extra '{\"sslmode\": \"require\"}'"
# Print the connection dictionary below
# Your code here

Hint

Use print(conn_postgres) to show the dictionary.

Practice

1. What is the main purpose of enabling connection encryption in Airflow?

easy

A. To protect data exchanged between Airflow and external systems

B. To speed up the connection between Airflow and databases

C. To reduce the size of data transferred over the network

D. To allow Airflow to connect without authentication

Solution

1. Step 1: Understand connection encryption

   Encryption secures data by making it unreadable to outsiders during transfer.

2. Step 2: Apply to Airflow connections

   Airflow uses encryption to protect sensitive data when communicating with external systems.

3. Final Answer:

   To protect data exchanged between Airflow and external systems -> Option A

4. Quick Check:

   Encryption protects data [OK]

Hint: Encryption means data is safe from outsiders [OK]

Common Mistakes:

• Thinking encryption speeds up connections
• Confusing encryption with data compression
• Assuming encryption removes authentication

2. Which of the following is the correct way to specify SSL encryption in an Airflow connection's extra field?

easy

A. {"sslmode": "require"}

B. {"ssl": "false"}

C. {"encrypt": "no"}

D. {"ssl_enabled": "off"}

Solution

1. Step 1: Identify correct SSL option key

   The standard key for SSL mode in Airflow connection extras is "sslmode" with values like "require".

2. Step 2: Check option values

   "require" enables SSL, while other options like "false", "no", or "off" disable encryption.

3. Final Answer:

   {"sslmode": "require"} -> Option A

4. Quick Check:

   Use "sslmode": "require" for SSL [OK]

Hint: Use "sslmode": "require" to enable SSL in extras [OK]

Common Mistakes:

• Using incorrect keys like "ssl_enabled"
• Setting SSL to false or no disables encryption
• Wrong JSON format in extras field

3. Given this Airflow connection extra field: {"sslmode": "verify-full", "sslrootcert": "/path/to/ca.pem"}, what is the effect when Airflow connects to the database?

medium

A. Airflow disables SSL and uses plain text

B. Airflow uses SSL but does not verify the server certificate

C. Airflow connects without encryption

D. Airflow uses SSL and verifies the server certificate against the CA file

Solution

1. Step 1: Understand sslmode "verify-full"

   This mode means SSL is used and the server certificate is fully verified.

2. Step 2: Role of sslrootcert

   The "sslrootcert" points to the CA certificate file used to verify the server's certificate authenticity.

3. Final Answer:

   Airflow uses SSL and verifies the server certificate against the CA file -> Option D

4. Quick Check:

   "verify-full" + CA cert = full SSL verification [OK]

Hint: "verify-full" means SSL with full certificate check [OK]

Common Mistakes:

• Confusing "verify-full" with no verification
• Ignoring the CA certificate path
• Assuming SSL is off with these settings

4. You configured an Airflow connection with {"sslmode": "require"} in the extra field, but Airflow still connects without encryption. What is the most likely cause?

medium

A. The database server does not support SSL connections

B. The Airflow scheduler was not restarted after the change

C. The extra field JSON is missing a closing brace

D. The connection type does not support SSL encryption

Solution

1. Step 1: Airflow connection caching

   Airflow components like the scheduler cache connections in memory. Changes to connections require restarting to load new configuration.

2. Step 2: Verify other causes

   Without restart, the new SSL extra is ignored, using cached unencrypted connection. Server lacking SSL support would fail the connection; malformed JSON or unsupported conn type would error differently.

3. Final Answer:

   The Airflow scheduler was not restarted after the change -> Option B

4. Quick Check:

   Restart Airflow after connection changes [OK]

Hint: Restart Airflow scheduler after connection config changes [OK]

Common Mistakes:

• Assuming Airflow ignores server capabilities
• Not checking server SSL configuration
• Forgetting to restart Airflow after config changes

5. You want to configure an Airflow connection to a PostgreSQL database with full SSL verification using a CA certificate file located at /certs/ca.pem. Which extra field JSON is correct?

hard

A. {"sslmode": "require", "sslrootcert": "/certs/ca.pem"}

B. {"sslmode": "disable", "sslrootcert": "/certs/ca.pem"}

C. {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"}

D. {"sslmode": "verify-ca", "sslrootcert": "/certs/ca.pem"}

Solution

1. Step 1: Understand SSL modes for PostgreSQL

   "verify-full" requires SSL and full verification of the server certificate against the CA file.

2. Step 2: Compare other modes

   "require" enables SSL but does not verify the server certificate fully; "disable" turns off SSL; "verify-ca" verifies the CA but not the hostname.

3. Final Answer:

   {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"} -> Option C

4. Quick Check:

   Full verification needs "verify-full" mode [OK]

Hint: "verify-full" + CA cert = strongest SSL verification [OK]

Common Mistakes:

• Using "require" instead of "verify-full" for full verification
• Confusing "verify-ca" with full hostname verification
• Disabling SSL accidentally