Connection encryption in Apache Airflow - Commands & Configuration

Introduction

Connection encryption protects data sent between Airflow and external services by making it unreadable to outsiders. This keeps sensitive information safe during communication.

When Airflow connects to a database that contains sensitive data and you want to keep the data safe while it travels over the network.

When Airflow communicates with cloud services like AWS or GCP and you want to ensure credentials and data are secure.

When you use Airflow to send emails or notifications and want to encrypt the connection to the mail server.

When Airflow connects to APIs that require secure HTTPS connections to protect data.

When you want to comply with security policies that require encrypted connections for all external communications.

Config File - airflow.cfg

airflow.cfg

[core]
# The connection string for the metadata database
sql_alchemy_conn = postgresql+psycopg2://airflow_user:[e-mail]:5432/airflow_db

[webserver]
# Enable HTTPS for the Airflow webserver
web_server_ssl_cert = /etc/ssl/certs/airflow.crt
web_server_ssl_key = /etc/ssl/private/airflow.key

[email]
# Use TLS encryption for SMTP email sending
smtp_host = smtp.example.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [e-mail]
smtp_password = emailpassword
smtp_port = 587

[core] section sets the database connection string with username and password to connect securely.

[webserver] section configures SSL certificates to enable HTTPS for the Airflow UI, encrypting browser traffic.

[email] section enables TLS encryption for sending emails securely through the SMTP server.

Commands

Initializes the Airflow metadata database using the encrypted connection string from airflow.cfg.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade head

INFO  [alembic.runtime.migration] Upgrade done.

Starts the Airflow webserver with SSL enabled to encrypt web traffic using the provided certificate and key files.

Terminal

airflow webserver --ssl-cert /etc/ssl/certs/airflow.crt --ssl-key /etc/ssl/private/airflow.key

Expected Output

Starting the web server on https://0.0.0.0:8080

INFO  - Listening on https://0.0.0.0:8080

INFO  - Worker started with pid 12345

→

--ssl-cert - Path to the SSL certificate file for HTTPS

→

--ssl-key - Path to the SSL private key file for HTTPS

Adds a connection in Airflow for the SMTP server with TLS encryption enabled for sending emails securely.

Terminal

airflow connections add 'smtp_default' --conn-uri 'smtp://[e-mail]:[e-mail]:587?starttls=True'

Expected Output

Added connection `smtp_default`

→

--conn-uri - Full connection URI including protocol, user, password, host, port, and TLS parameter

Verifies the SMTP connection details including encryption settings are stored correctly in Airflow.

Terminal

airflow connections get 'smtp_default'

Expected Output

Conn Id: smtp_default

Conn Type: smtp

Host: smtp.example.com

Login: [e-mail]

Port: 587

Extra: {"starttls": true}

Key Concept

If you remember nothing else from this pattern, remember: encrypting connections keeps your data safe when Airflow talks to other services.

Common Mistakes

Not enabling TLS or SSL in the connection settings

Data sent over the network can be intercepted and read by attackers if encryption is off.

Always set TLS or SSL options in your connection strings or configuration files when supported.

Using self-signed certificates without trusting them in Airflow or the browser

The connection will fail or show security warnings, preventing secure communication.

Use certificates signed by a trusted authority or configure Airflow and clients to trust your self-signed certificates.

Hardcoding passwords in plain text without encryption or secrets management

Passwords can be exposed if the config file or environment is accessed by unauthorized users.

Use Airflow's secrets backend or environment variables to store sensitive credentials securely.

Summary

Configure Airflow connections and services to use TLS/SSL for encrypting data in transit.

Use airflow.cfg and connection URIs to specify encryption settings like TLS for SMTP and HTTPS for the webserver.

Verify encryption by checking connection details and running Airflow commands that use these secure connections.

Practice

1. What is the main purpose of enabling connection encryption in Airflow?

easy

A. To protect data exchanged between Airflow and external systems

B. To speed up the connection between Airflow and databases

C. To reduce the size of data transferred over the network

D. To allow Airflow to connect without authentication

Solution

1. Step 1: Understand connection encryption

   Encryption secures data by making it unreadable to outsiders during transfer.

2. Step 2: Apply to Airflow connections

   Airflow uses encryption to protect sensitive data when communicating with external systems.

3. Final Answer:

   To protect data exchanged between Airflow and external systems -> Option A

4. Quick Check:

   Encryption protects data [OK]

Hint: Encryption means data is safe from outsiders [OK]

Common Mistakes:

• Thinking encryption speeds up connections
• Confusing encryption with data compression
• Assuming encryption removes authentication

2. Which of the following is the correct way to specify SSL encryption in an Airflow connection's extra field?

easy

A. {"sslmode": "require"}

B. {"ssl": "false"}

C. {"encrypt": "no"}

D. {"ssl_enabled": "off"}

Solution

1. Step 1: Identify correct SSL option key

   The standard key for SSL mode in Airflow connection extras is "sslmode" with values like "require".

2. Step 2: Check option values

   "require" enables SSL, while other options like "false", "no", or "off" disable encryption.

3. Final Answer:

   {"sslmode": "require"} -> Option A

4. Quick Check:

   Use "sslmode": "require" for SSL [OK]

Hint: Use "sslmode": "require" to enable SSL in extras [OK]

Common Mistakes:

• Using incorrect keys like "ssl_enabled"
• Setting SSL to false or no disables encryption
• Wrong JSON format in extras field

3. Given this Airflow connection extra field: {"sslmode": "verify-full", "sslrootcert": "/path/to/ca.pem"}, what is the effect when Airflow connects to the database?

medium

A. Airflow disables SSL and uses plain text

B. Airflow uses SSL but does not verify the server certificate

C. Airflow connects without encryption

D. Airflow uses SSL and verifies the server certificate against the CA file

Solution

1. Step 1: Understand sslmode "verify-full"

   This mode means SSL is used and the server certificate is fully verified.

2. Step 2: Role of sslrootcert

   The "sslrootcert" points to the CA certificate file used to verify the server's certificate authenticity.

3. Final Answer:

   Airflow uses SSL and verifies the server certificate against the CA file -> Option D

4. Quick Check:

   "verify-full" + CA cert = full SSL verification [OK]

Hint: "verify-full" means SSL with full certificate check [OK]

Common Mistakes:

• Confusing "verify-full" with no verification
• Ignoring the CA certificate path
• Assuming SSL is off with these settings

4. You configured an Airflow connection with {"sslmode": "require"} in the extra field, but Airflow still connects without encryption. What is the most likely cause?

medium

A. The database server does not support SSL connections

B. The Airflow scheduler was not restarted after the change

C. The extra field JSON is missing a closing brace

D. The connection type does not support SSL encryption

Solution

1. Step 1: Airflow connection caching

   Airflow components like the scheduler cache connections in memory. Changes to connections require restarting to load new configuration.

2. Step 2: Verify other causes

   Without restart, the new SSL extra is ignored, using cached unencrypted connection. Server lacking SSL support would fail the connection; malformed JSON or unsupported conn type would error differently.

3. Final Answer:

   The Airflow scheduler was not restarted after the change -> Option B

4. Quick Check:

   Restart Airflow after connection changes [OK]

Hint: Restart Airflow scheduler after connection config changes [OK]

Common Mistakes:

• Assuming Airflow ignores server capabilities
• Not checking server SSL configuration
• Forgetting to restart Airflow after config changes

5. You want to configure an Airflow connection to a PostgreSQL database with full SSL verification using a CA certificate file located at /certs/ca.pem. Which extra field JSON is correct?

hard

A. {"sslmode": "require", "sslrootcert": "/certs/ca.pem"}

B. {"sslmode": "disable", "sslrootcert": "/certs/ca.pem"}

C. {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"}

D. {"sslmode": "verify-ca", "sslrootcert": "/certs/ca.pem"}

Solution

1. Step 1: Understand SSL modes for PostgreSQL

   "verify-full" requires SSL and full verification of the server certificate against the CA file.

2. Step 2: Compare other modes

   "require" enables SSL but does not verify the server certificate fully; "disable" turns off SSL; "verify-ca" verifies the CA but not the hostname.

3. Final Answer:

   {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"} -> Option C

4. Quick Check:

   Full verification needs "verify-full" mode [OK]

Hint: "verify-full" + CA cert = strongest SSL verification [OK]

Common Mistakes:

• Using "require" instead of "verify-full" for full verification
• Confusing "verify-ca" with full hostname verification
• Disabling SSL accidentally