Authentication backends (LDAP, OAuth) in Apache Airflow - Mini Project: Build & Apply

Configure Authentication Backends in Apache Airflow

📖 Scenario: You are setting up Apache Airflow for your team. To keep it secure, you want to enable user login using an external authentication system. Your company supports LDAP and OAuth for user authentication.This project will guide you through configuring Airflow to use these authentication backends step-by-step.

🎯 Goal: By the end, you will have configured Airflow's airflow.cfg file to enable LDAP and OAuth authentication backends. You will also verify the configuration by printing the active authentication backend setting.

📋 What You'll Learn

Use the exact variable names and values as instructed

Modify the airflow.cfg configuration settings as text variables

Print the final authentication backend setting exactly as specified

💡 Why This Matters

🌍 Real World

Many companies use Apache Airflow to schedule and monitor workflows. Securing access with LDAP or OAuth is common to protect sensitive data and operations.

💼 Career

Understanding how to configure authentication backends in Airflow is important for DevOps engineers and data engineers managing secure workflow environments.

1

Create initial Airflow configuration dictionary

Create a dictionary called airflow_cfg with one key "webserver_config" set to an empty string.

Apache Airflow

# Create the airflow_cfg dictionary with 'webserver_config' key
# Your code here

Hint

Use curly braces {} to create a dictionary and set the key 'webserver_config' to an empty string "".

2

Add LDAP authentication backend setting

Add a new key "auth_backend" to the airflow_cfg dictionary and set its value to "airflow.contrib.auth.backends.ldap_auth".

Apache Airflow

airflow_cfg = {"webserver_config": ""}
# Add 'auth_backend' key with LDAP backend value
# Your code here

Hint

Use the exact key 'auth_backend' and set it to the LDAP backend string.

3

Switch authentication backend to OAuth

Change the value of the "auth_backend" key in airflow_cfg to "airflow.contrib.auth.backends.oauth_auth".

Apache Airflow

airflow_cfg = {"webserver_config": "", "auth_backend": "airflow.contrib.auth.backends.ldap_auth"}
# Change 'auth_backend' value to OAuth backend
# Your code here

Hint

Assign the OAuth backend string exactly to the 'auth_backend' key.

4

Print the active authentication backend

Write a print statement to display the value of the "auth_backend" key from the airflow_cfg dictionary.

Apache Airflow

airflow_cfg = {"webserver_config": "", "auth_backend": "airflow.contrib.auth.backends.oauth_auth"}
# Print the current auth_backend value
# Your code here

Hint

Use print(airflow_cfg["auth_backend"]) to show the current authentication backend.

Practice

1. What is the main purpose of configuring an authentication backend like LDAP or OAuth in Airflow?

easy

A. To schedule DAGs automatically

B. To verify users using external systems for better security

C. To store Airflow logs externally

D. To speed up Airflow task execution

Solution

1. Step 1: Understand authentication backend role

   Authentication backends connect Airflow to external user verification systems like LDAP or OAuth.

2. Step 2: Identify the main benefit

   This setup improves security and user management by using existing accounts outside Airflow.

3. Final Answer:

   To verify users using external systems for better security -> Option B

4. Quick Check:

   Authentication backend = user verification [OK]

Hint: Authentication backends verify users externally [OK]

Common Mistakes:

• Confusing authentication with task scheduling
• Thinking it speeds up task execution
• Assuming it stores logs

2. Which of the following is the correct way to specify the LDAP authentication backend in the airflow.cfg file?

easy

A. auth_backend = airflow.providers.ldap.auth.backend.LDAPAuthBackend

B. auth_backend = airflow.contrib.auth.backends.ldap_auth

C. auth_backend = airflow.www.auth.backend.ldap

D. auth_backend = airflow.contrib.auth.backends.ldap_auth_backend.LDAPBackend

Solution

1. Step 1: Recall the correct LDAP backend path

   The correct LDAP backend path in Airflow 2.x is airflow.providers.ldap.auth.backend.LDAPAuthBackend.

2. Step 2: Compare options

   auth_backend = airflow.providers.ldap.auth.backend.LDAPAuthBackend matches the correct syntax exactly; others have incorrect module paths or class names.

3. Final Answer:

   auth_backend = airflow.providers.ldap.auth.backend.LDAPAuthBackend -> Option A

4. Quick Check:

   Correct LDAP backend path = auth_backend = airflow.providers.ldap.auth.backend.LDAPAuthBackend [OK]

Hint: Use airflow.providers.ldap.auth.backend.LDAPAuthBackend for LDAP [OK]

Common Mistakes:

• Using incorrect module paths
• Adding extra class names
• Confusing OAuth backend with LDAP

3. Given this snippet in airflow.cfg:

[webserver]
auth_backend = airflow.contrib.auth.backends.oauth_auth

What will happen when a user tries to log in?

medium

A. Airflow will allow login without authentication

B. Airflow will reject all logins because OAuth is not supported

C. Airflow will use LDAP authentication instead

D. Airflow will authenticate the user using OAuth provider settings

Solution

1. Step 1: Identify the configured backend

   The config sets auth_backend to airflow.contrib.auth.backends.oauth_auth, which enables OAuth authentication.

2. Step 2: Understand login behavior

   With OAuth backend, Airflow redirects login to the OAuth provider for user verification.

3. Final Answer:

   Airflow will authenticate the user using OAuth provider settings -> Option D

4. Quick Check:

   OAuth backend means OAuth login [OK]

Hint: OAuth backend means login via OAuth provider [OK]

Common Mistakes:

• Assuming OAuth is unsupported
• Confusing OAuth with LDAP
• Thinking login is disabled

4. You configured LDAP authentication in airflow.cfg but users cannot log in. Which of these is the most likely cause?

medium

A. The Airflow scheduler is not running

B. The DAGs folder is empty

C. The auth_backend setting points to a non-existent module

D. The webserver port is set to 8080

Solution

1. Step 1: Analyze login failure causes

   If users cannot log in after LDAP setup, a common cause is a wrong or missing auth_backend module path.

2. Step 2: Eliminate unrelated causes

   Empty DAGs, scheduler status, or port settings do not affect authentication backend functionality.

3. Final Answer:

   The auth_backend setting points to a non-existent module -> Option C

4. Quick Check:

   Wrong auth_backend module = login failure [OK]

Hint: Check auth_backend module path if login fails [OK]

Common Mistakes:

• Blaming DAGs folder for login issues
• Confusing scheduler status with auth problems
• Thinking port affects authentication

5. You want Airflow to authenticate users via OAuth and also allow admin users to log in with LDAP. Which approach is best to achieve this?

hard

A. Configure a custom authentication backend that combines OAuth and LDAP logic

B. Set auth_backend to LDAP only and ignore OAuth

C. Use OAuth backend and manually add admin users in Airflow database

D. Run two Airflow instances, one with LDAP and one with OAuth

Solution

1. Step 1: Understand multi-backend authentication needs

   Airflow supports one auth_backend at a time, so combining OAuth and LDAP requires custom code.

2. Step 2: Evaluate options

   Configure a custom authentication backend that combines OAuth and LDAP logic suggests creating a custom backend to handle both OAuth and LDAP, which is the correct approach.

3. Step 3: Reject other options

   Setting auth_backend to LDAP only, using OAuth backend and manually adding admin users, or running two Airflow instances do not support both methods simultaneously; the latter is also complex and inefficient.

4. Final Answer:

   Configure a custom authentication backend that combines OAuth and LDAP logic -> Option A

5. Quick Check:

   Multi-auth needs custom backend [OK]

Hint: Combine OAuth and LDAP with custom backend code [OK]

Common Mistakes:

• Trying to set two auth_backends in airflow.cfg
• Ignoring admin user needs
• Running multiple Airflow instances unnecessarily