Connection encryption in Apache Airflow - Commands & Configuration

Introduction

Connection encryption protects data sent between Airflow and external services by making it unreadable to outsiders. This keeps sensitive information safe during communication.

When Airflow connects to a database that contains sensitive data and you want to keep the data safe while it travels over the network.

When Airflow communicates with cloud services like AWS or GCP and you want to ensure credentials and data are secure.

When you use Airflow to send emails or notifications and want to encrypt the connection to the mail server.

When Airflow connects to APIs that require secure HTTPS connections to protect data.

When you want to comply with security policies that require encrypted connections for all external communications.

Config File - airflow.cfg

airflow.cfg

[core]
# The connection string for the metadata database
sql_alchemy_conn = postgresql+psycopg2://airflow_user:[e-mail]:5432/airflow_db

[webserver]
# Enable HTTPS for the Airflow webserver
web_server_ssl_cert = /etc/ssl/certs/airflow.crt
web_server_ssl_key = /etc/ssl/private/airflow.key

[email]
# Use TLS encryption for SMTP email sending
smtp_host = smtp.example.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [e-mail]
smtp_password = emailpassword
smtp_port = 587

[core] section sets the database connection string with username and password to connect securely.

[webserver] section configures SSL certificates to enable HTTPS for the Airflow UI, encrypting browser traffic.

[email] section enables TLS encryption for sending emails securely through the SMTP server.

Commands

Initializes the Airflow metadata database using the encrypted connection string from airflow.cfg.

Terminal

airflow db init

Expected Output

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.

INFO  [alembic.runtime.migration] Will assume transactional DDL.

INFO  [alembic.runtime.migration] Running upgrade head

INFO  [alembic.runtime.migration] Upgrade done.

Starts the Airflow webserver with SSL enabled to encrypt web traffic using the provided certificate and key files.

Terminal

airflow webserver --ssl-cert /etc/ssl/certs/airflow.crt --ssl-key /etc/ssl/private/airflow.key

Expected Output

Starting the web server on https://0.0.0.0:8080

INFO  - Listening on https://0.0.0.0:8080

INFO  - Worker started with pid 12345

→

--ssl-cert - Path to the SSL certificate file for HTTPS

→

--ssl-key - Path to the SSL private key file for HTTPS

Adds a connection in Airflow for the SMTP server with TLS encryption enabled for sending emails securely.

Terminal

airflow connections add 'smtp_default' --conn-uri 'smtp://[e-mail]:[e-mail]:587?starttls=True'

Expected Output

Added connection `smtp_default`

→

--conn-uri - Full connection URI including protocol, user, password, host, port, and TLS parameter

Verifies the SMTP connection details including encryption settings are stored correctly in Airflow.

Terminal

airflow connections get 'smtp_default'

Expected Output

Conn Id: smtp_default

Conn Type: smtp

Host: smtp.example.com

Login: [e-mail]

Port: 587

Extra: {"starttls": true}

Key Concept

If you remember nothing else from this pattern, remember: encrypting connections keeps your data safe when Airflow talks to other services.

Common Mistakes

Not enabling TLS or SSL in the connection settings

Data sent over the network can be intercepted and read by attackers if encryption is off.

Always set TLS or SSL options in your connection strings or configuration files when supported.

Using self-signed certificates without trusting them in Airflow or the browser

The connection will fail or show security warnings, preventing secure communication.

Use certificates signed by a trusted authority or configure Airflow and clients to trust your self-signed certificates.

Hardcoding passwords in plain text without encryption or secrets management

Passwords can be exposed if the config file or environment is accessed by unauthorized users.

Use Airflow's secrets backend or environment variables to store sensitive credentials securely.

Summary

Configure Airflow connections and services to use TLS/SSL for encrypting data in transit.

Use airflow.cfg and connection URIs to specify encryption settings like TLS for SMTP and HTTPS for the webserver.

Verify encryption by checking connection details and running Airflow commands that use these secure connections.