Connection encryption in Apache Airflow - Step-by-Step Execution

Process Flow - Connection encryption

Start Airflow Connection Setup

↓

Define Connection Parameters

↓

Enable Encryption Settings?

No→Use Plain Connection

Yes↓

Set Encryption Protocol (e.g., TLS)

↓

Provide Certificates/Keys

↓

Test Connection with Encryption

↓

Connection Established Securely

↓

End

This flow shows how Airflow sets up a connection with encryption by enabling encryption, setting protocols, providing keys, and testing the secure connection.

Execution Sample

Apache Airflow

conn = Connection(
    conn_id='my_db',
    conn_type='postgres',
    host='db.example.com',
    login='user',
    password='pass',
    extra='{"sslmode": "require"}'
)
session.add(conn)
session.commit()

This code creates a new Airflow connection to a Postgres database with SSL encryption required.

Process Table

Step	Action	Parameter Set	Encryption Enabled	Result
1	Create Connection object	conn_id='my_db', conn_type='postgres', host='db.example.com', login='user', password='pass'	No	Connection object created without encryption
2	Add extra parameter for SSL	extra='{"sslmode": "require"}'	Yes	Encryption flag set to require SSL
3	Add connection to session	session.add(conn)	Yes	Connection staged for saving
4	Commit session	session.commit()	Yes	Connection saved with encryption settings
5	Test connection	Connect to db.example.com with SSL	Yes	Connection established securely
6	End	-	-	Setup complete with encryption

💡 Connection saved and tested with SSL encryption enabled

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
conn	None	Connection object without extra	Connection object with extra SSL param	Same	Same	Saved in Airflow DB with SSL
encryption_enabled	False	False	True	True	True	True

Key Moments - 2 Insights

Why do we add the 'extra' parameter with sslmode instead of a direct attribute?

Airflow uses the 'extra' JSON field to pass additional connection options like SSL settings. This is shown in execution_table step 2 where adding 'extra' enables encryption.

What happens if we skip the commit step after adding the connection?

Without committing (step 4), the connection is not saved in Airflow's database, so encryption settings won't persist. The connection won't be usable.

Visual Quiz - 3 Questions

Look at the execution_table at step 2, what change happens to the connection?

A Connection is deleted

B Connection type changes to MySQL

C Encryption is enabled by adding SSL parameters

D Password is removed

💡 Hint

Refer to the 'Encryption Enabled' and 'Parameter Set' columns at step 2 in execution_table

At which step is the connection saved permanently in Airflow's database?

A Step 1

B Step 4

C Step 3

D Step 5

💡 Hint

Look at the 'Result' column in execution_table for the commit action

If we remove the 'extra' parameter, what will be the encryption status?

A Encryption disabled

B Encryption enabled

C Connection fails to create

D Encryption status unknown

💡 Hint

Check variable_tracker for 'encryption_enabled' after step 1 and 2

Concept Snapshot

Airflow Connection Encryption Setup:
- Create Connection object with basic params
- Use 'extra' JSON to add SSL (e.g., 'sslmode':'require')
- Add and commit connection to Airflow DB
- Test connection to confirm encryption
- Encryption ensures data security during connection

Full Transcript

This visual execution shows how to set up an encrypted connection in Airflow. First, a Connection object is created with basic details like host and login. Then, encryption is enabled by adding SSL parameters inside the 'extra' JSON field. The connection is added to the session and committed to save it permanently. Finally, the connection is tested to confirm it uses encryption. Variables like 'conn' and 'encryption_enabled' track the state changes. Key moments clarify why the 'extra' field is used and the importance of committing. Quizzes check understanding of when encryption is enabled and saved.

Practice

1. What is the main purpose of enabling connection encryption in Airflow?

easy

A. To protect data exchanged between Airflow and external systems

B. To speed up the connection between Airflow and databases

C. To reduce the size of data transferred over the network

D. To allow Airflow to connect without authentication

Solution

1. Step 1: Understand connection encryption

   Encryption secures data by making it unreadable to outsiders during transfer.

2. Step 2: Apply to Airflow connections

   Airflow uses encryption to protect sensitive data when communicating with external systems.

3. Final Answer:

   To protect data exchanged between Airflow and external systems -> Option A

4. Quick Check:

   Encryption protects data [OK]

Hint: Encryption means data is safe from outsiders [OK]

Common Mistakes:

• Thinking encryption speeds up connections
• Confusing encryption with data compression
• Assuming encryption removes authentication

2. Which of the following is the correct way to specify SSL encryption in an Airflow connection's extra field?

easy

A. {"sslmode": "require"}

B. {"ssl": "false"}

C. {"encrypt": "no"}

D. {"ssl_enabled": "off"}

Solution

1. Step 1: Identify correct SSL option key

   The standard key for SSL mode in Airflow connection extras is "sslmode" with values like "require".

2. Step 2: Check option values

   "require" enables SSL, while other options like "false", "no", or "off" disable encryption.

3. Final Answer:

   {"sslmode": "require"} -> Option A

4. Quick Check:

   Use "sslmode": "require" for SSL [OK]

Hint: Use "sslmode": "require" to enable SSL in extras [OK]

Common Mistakes:

• Using incorrect keys like "ssl_enabled"
• Setting SSL to false or no disables encryption
• Wrong JSON format in extras field

3. Given this Airflow connection extra field: {"sslmode": "verify-full", "sslrootcert": "/path/to/ca.pem"}, what is the effect when Airflow connects to the database?

medium

A. Airflow disables SSL and uses plain text

B. Airflow uses SSL but does not verify the server certificate

C. Airflow connects without encryption

D. Airflow uses SSL and verifies the server certificate against the CA file

Solution

1. Step 1: Understand sslmode "verify-full"

   This mode means SSL is used and the server certificate is fully verified.

2. Step 2: Role of sslrootcert

   The "sslrootcert" points to the CA certificate file used to verify the server's certificate authenticity.

3. Final Answer:

   Airflow uses SSL and verifies the server certificate against the CA file -> Option D

4. Quick Check:

   "verify-full" + CA cert = full SSL verification [OK]

Hint: "verify-full" means SSL with full certificate check [OK]

Common Mistakes:

• Confusing "verify-full" with no verification
• Ignoring the CA certificate path
• Assuming SSL is off with these settings

4. You configured an Airflow connection with {"sslmode": "require"} in the extra field, but Airflow still connects without encryption. What is the most likely cause?

medium

A. The database server does not support SSL connections

B. The Airflow scheduler was not restarted after the change

C. The extra field JSON is missing a closing brace

D. The connection type does not support SSL encryption

Solution

1. Step 1: Airflow connection caching

   Airflow components like the scheduler cache connections in memory. Changes to connections require restarting to load new configuration.

2. Step 2: Verify other causes

   Without restart, the new SSL extra is ignored, using cached unencrypted connection. Server lacking SSL support would fail the connection; malformed JSON or unsupported conn type would error differently.

3. Final Answer:

   The Airflow scheduler was not restarted after the change -> Option B

4. Quick Check:

   Restart Airflow after connection changes [OK]

Hint: Restart Airflow scheduler after connection config changes [OK]

Common Mistakes:

• Assuming Airflow ignores server capabilities
• Not checking server SSL configuration
• Forgetting to restart Airflow after config changes

5. You want to configure an Airflow connection to a PostgreSQL database with full SSL verification using a CA certificate file located at /certs/ca.pem. Which extra field JSON is correct?

hard

A. {"sslmode": "require", "sslrootcert": "/certs/ca.pem"}

B. {"sslmode": "disable", "sslrootcert": "/certs/ca.pem"}

C. {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"}

D. {"sslmode": "verify-ca", "sslrootcert": "/certs/ca.pem"}

Solution

1. Step 1: Understand SSL modes for PostgreSQL

   "verify-full" requires SSL and full verification of the server certificate against the CA file.

2. Step 2: Compare other modes

   "require" enables SSL but does not verify the server certificate fully; "disable" turns off SSL; "verify-ca" verifies the CA but not the hostname.

3. Final Answer:

   {"sslmode": "verify-full", "sslrootcert": "/certs/ca.pem"} -> Option C

4. Quick Check:

   Full verification needs "verify-full" mode [OK]

Hint: "verify-full" + CA cert = strongest SSL verification [OK]

Common Mistakes:

• Using "require" instead of "verify-full" for full verification
• Confusing "verify-ca" with full hostname verification
• Disabling SSL accidentally