Multi-environment deployment (dev, staging, prod) in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use separate Airflow environments for dev, staging, and prod?

Which of the following is the main reason to maintain separate Airflow environments for development, staging, and production?

A To increase the number of users who can access Airflow UI simultaneously

B To isolate changes and test workflows safely before affecting production data

C To reduce the number of DAGs that can be deployed

D To save disk space by sharing logs across environments

💡 Hint

Think about why testing in a safe place before production is important.

Separate environments allow developers to test changes without risking production stability or data integrity.

💻 Command Output

intermediate

Output of Airflow environment variable check

Given the following command run in the staging Airflow environment, what is the output?

echo $AIRFLOW_ENV

A staging

B production

C development

D undefined

💡 Hint

Check which environment variable is set for staging.

The AIRFLOW_ENV variable is set to 'staging' in the staging environment to distinguish it from dev and prod.

❓ Configuration

advanced

Correct Airflow config for environment-specific database URI

Which Airflow configuration snippet correctly sets the database URI for the production environment using environment variables?

A

[database]
sql_alchemy_conn = $PROD_DB_URI

B

[database]
sql_alchemy_conn = PROD_DB_URI

C

[database]
sql_alchemy_conn = ${PROD_DB_URI}

D

[database]
dsql_alchemy_conn = ${PROD_DB_URI}

💡 Hint

Check the correct config key spelling and environment variable syntax.

The correct key is 'sql_alchemy_conn' and environment variables are referenced with ${VAR} or $VAR in some shells, but Airflow config uses ${VAR} syntax inside the config file.

🔀 Workflow

advanced

Best practice for deploying DAGs across environments

Which deployment workflow ensures safe promotion of DAGs from development to production in Airflow?

A Directly commit DAGs to the production repository branch and deploy

B Use a single branch for all environments and toggle environment variables manually

C Deploy DAGs only in production and fix bugs live

D Develop DAGs locally, push to a dev branch, test in dev environment, merge to staging branch for further testing, then merge to production branch for deployment

💡 Hint

Think about how to test changes progressively before production.

This workflow uses branches and environments to test changes step-by-step, reducing risk of errors in production.

❓ Troubleshoot

expert

Why does the production Airflow environment fail to connect to the database?

The production Airflow environment suddenly fails to connect to its database. The error log shows: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server. The sql_alchemy_conn in airflow.cfg is set to ${PROD_DB_URI}. What is the most likely cause?

A The environment variable PROD_DB_URI is not set or empty in the production environment

B The Airflow scheduler is not running

C The DAGs folder is empty

D The Airflow webserver port is blocked by firewall

💡 Hint

Check if the environment variable used in the config is properly set.

If PROD_DB_URI is missing or empty, Airflow cannot connect to the database, causing the OperationalError.

Practice

1. What is the main reason to use separate environments like dev, staging, and prod in Airflow deployment?

easy

A. To run all tasks faster by using multiple servers

B. To reduce the number of DAGs needed

C. To keep development, testing, and production work isolated and safe

D. To share the same database for all environments

Solution

1. Step 1: Understand environment separation purpose

   Separate environments prevent changes in development or testing from affecting production stability.

2. Step 2: Identify the correct purpose in options

   Only To keep development, testing, and production work isolated and safe mentions isolation and safety, which matches the main reason.

3. Final Answer:

   To keep development, testing, and production work isolated and safe -> Option C

4. Quick Check:

   Environment isolation = Safety [OK]

Hint: Remember: separate environments protect production [OK]

Common Mistakes:

• Thinking all environments share the same database
• Believing environments speed up task execution
• Assuming fewer DAGs are needed with multiple environments

2. Which of the following is the correct way to specify different Airflow configurations for dev and prod environments?

easy

A. Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment

B. Run Airflow without any configuration files

C. Use the same airflow.cfg file for all environments

D. Use a single database for all environments

Solution

1. Step 1: Understand configuration methods in Airflow

   Airflow supports environment variables to override settings per environment.

2. Step 2: Identify the correct method for environment-specific configs

   Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment correctly uses environment variables to set different database connections per environment.

3. Final Answer:

   Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment -> Option A

4. Quick Check:

   Env vars = environment-specific config [OK]

Hint: Use env vars to separate configs per environment [OK]

Common Mistakes:

• Using the same config file for all environments
• Not setting database connections separately
• Ignoring environment variables for configs

3. Given this command to start Airflow in the staging environment:

export AIRFLOW_HOME=/opt/airflow_staging
export AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://user:pass@staging-db:5432/airflow
airflow webserver

What will happen when you run this command?

medium

A. Airflow will start using the staging database and configs in /opt/airflow_staging

B. Airflow will start using the production database and default configs

C. Airflow will fail because environment variables cannot be set this way

D. Airflow will start but ignore the SQL connection string

Solution

1. Step 1: Analyze environment variables set before starting Airflow

   Setting AIRFLOW_HOME changes the config directory; setting AIRFLOW__CORE__SQL_ALCHEMY_CONN changes the database connection.

2. Step 2: Understand Airflow behavior on startup

   Airflow uses these environment variables to load configs and connect to the specified database.

3. Final Answer:

   Airflow will start using the staging database and configs in /opt/airflow_staging -> Option A

4. Quick Check:

   Env vars control config and DB connection [OK]

Hint: Env vars before airflow start set environment [OK]

Common Mistakes:

• Assuming Airflow ignores env vars
• Thinking Airflow uses production DB by default
• Believing env vars cannot be set inline

4. You deployed Airflow for prod but accidentally used the dev database connection string in the config. What is the best way to fix this?

medium

A. Change the airflow.cfg file in the dev environment

B. Delete all DAGs and restart Airflow

C. Run Airflow without any environment variables

D. Change the AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable to the production database and restart Airflow

Solution

1. Step 1: Identify the incorrect database connection setting

   The problem is the wrong database connection string used in production.

2. Step 2: Correct the environment variable and restart Airflow

   Updating AIRFLOW__CORE__SQL_ALCHEMY_CONN to the correct production DB and restarting applies the fix.

3. Final Answer:

   Change the AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable to the production database and restart Airflow -> Option D

4. Quick Check:

   Fix DB connection env var and restart [OK]

Hint: Fix DB env var, then restart Airflow [OK]

Common Mistakes:

• Deleting DAGs unnecessarily
• Ignoring environment variables
• Changing dev configs instead of prod

5. You want to deploy Airflow for dev, staging, and prod on the same server. Which approach ensures proper isolation and avoids conflicts?

hard

A. Run one Airflow instance and switch configs manually before each run

B. Run three separate Airflow instances with different AIRFLOW_HOME and database connections

C. Use the same database but different DAG folders for each environment

D. Run Airflow only in production mode and test DAGs there

Solution

1. Step 1: Understand isolation needs for multiple environments on one server

   Each environment needs its own config directory and database to avoid conflicts.

2. Step 2: Identify the correct deployment approach

   Running separate Airflow instances with distinct AIRFLOW_HOME and DB connections ensures isolation.

3. Final Answer:

   Run three separate Airflow instances with different AIRFLOW_HOME and database connections -> Option B

4. Quick Check:

   Separate instances + configs = isolation [OK]

Hint: Use separate AIRFLOW_HOME and DB per instance [OK]

Common Mistakes:

• Switching configs manually risks errors
• Sharing one database causes conflicts
• Running only production ignores testing needs