Multi-environment deployment (dev, staging, prod) in Apache Airflow - Step-by-Step Execution

Process Flow - Multi-environment deployment (dev, staging, prod)

Write DAG code

↓

Set environment variables

↓

Configure airflow.cfg or use variables

↓

Deploy DAG to environment folder

↓

Airflow scheduler picks DAG

↓

Run DAG with environment-specific configs

↓

Monitor logs and outputs

↓

Repeat for dev, staging, prod

This flow shows how the same Airflow DAG code is deployed to different environments with specific settings, allowing safe testing and production runs.

Execution Sample

Apache Airflow

import os
from airflow import DAG
from airflow.operators.python import PythonOperator

def print_env():
    print(f"Running in {os.getenv('ENV')} environment")

This DAG prints the current environment by reading the ENV variable, showing how environment affects execution.

Process Table

Step	Action	Environment Variable ENV	DAG Behavior	Output
1	Set ENV=dev	dev	DAG reads ENV=dev	Prints: Running in dev environment
2	Deploy DAG to dev folder	dev	Scheduler picks DAG	DAG ready in dev
3	Run DAG in dev	dev	print_env runs	Running in dev environment
4	Set ENV=staging	staging	DAG reads ENV=staging	Prints: Running in staging environment
5	Deploy DAG to staging folder	staging	Scheduler picks DAG	DAG ready in staging
6	Run DAG in staging	staging	print_env runs	Running in staging environment
7	Set ENV=prod	prod	DAG reads ENV=prod	Prints: Running in prod environment
8	Deploy DAG to prod folder	prod	Scheduler picks DAG	DAG ready in prod
9	Run DAG in prod	prod	print_env runs	Running in prod environment
10	End	-	-	All environments deployed and tested

💡 All environments have been deployed and DAGs executed with their specific ENV variable.

Status Tracker

Variable	Start	After Step 1	After Step 4	After Step 7	Final
ENV	undefined	dev	staging	prod	prod

Key Moments - 3 Insights

Why does the DAG print different environment names even though the code is the same?

Because the ENV variable changes before deployment (see execution_table steps 1,4,7), the DAG reads this variable at runtime and behaves accordingly.

What happens if you forget to set the ENV variable before deploying?

The DAG will print 'Running in None environment' or error because ENV is not set, as shown by the variable_tracker start value.

Why do we deploy the same DAG code to different folders?

To isolate environments so changes in dev or staging don't affect prod, as shown in execution_table steps 2,5,8.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of ENV when the DAG runs in staging?

A staging

B dev

C prod

D undefined

💡 Hint

Check the 'Environment Variable ENV' column at step 6 where DAG runs in staging.

At which step does the DAG first print 'Running in prod environment'?

A Step 7

B Step 8

C Step 9

D Step 10

💡 Hint

Look at the 'Output' column for the prod environment print statement.

If ENV was never set, what would the DAG output be when run?

A Running in dev environment

B Running in None environment

C Error: ENV not found

D Running in prod environment

💡 Hint

Refer to variable_tracker start value and how print_env uses os.getenv('ENV').

Concept Snapshot

Multi-environment deployment in Airflow:
- Use same DAG code for dev, staging, prod
- Set ENV variable differently per environment
- Deploy DAGs to separate folders
- DAG reads ENV at runtime to adjust behavior
- Allows safe testing before production run

Full Transcript

This visual execution shows how to deploy the same Airflow DAG code to multiple environments: development, staging, and production. We set an environment variable called ENV to identify the environment. The DAG reads this variable at runtime and prints which environment it is running in. We deploy the DAG to separate folders for each environment so they do not interfere. The scheduler picks up the DAG in each environment and runs it with the correct settings. This approach helps test changes safely in dev and staging before deploying to production.

Practice

1. What is the main reason to use separate environments like dev, staging, and prod in Airflow deployment?

easy

A. To run all tasks faster by using multiple servers

B. To reduce the number of DAGs needed

C. To keep development, testing, and production work isolated and safe

D. To share the same database for all environments

Solution

1. Step 1: Understand environment separation purpose

   Separate environments prevent changes in development or testing from affecting production stability.

2. Step 2: Identify the correct purpose in options

   Only To keep development, testing, and production work isolated and safe mentions isolation and safety, which matches the main reason.

3. Final Answer:

   To keep development, testing, and production work isolated and safe -> Option C

4. Quick Check:

   Environment isolation = Safety [OK]

Hint: Remember: separate environments protect production [OK]

Common Mistakes:

• Thinking all environments share the same database
• Believing environments speed up task execution
• Assuming fewer DAGs are needed with multiple environments

2. Which of the following is the correct way to specify different Airflow configurations for dev and prod environments?

easy

A. Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment

B. Run Airflow without any configuration files

C. Use the same airflow.cfg file for all environments

D. Use a single database for all environments

Solution

1. Step 1: Understand configuration methods in Airflow

   Airflow supports environment variables to override settings per environment.

2. Step 2: Identify the correct method for environment-specific configs

   Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment correctly uses environment variables to set different database connections per environment.

3. Final Answer:

   Set environment variables like AIRFLOW__CORE__SQL_ALCHEMY_CONN differently per environment -> Option A

4. Quick Check:

   Env vars = environment-specific config [OK]

Hint: Use env vars to separate configs per environment [OK]

Common Mistakes:

• Using the same config file for all environments
• Not setting database connections separately
• Ignoring environment variables for configs

3. Given this command to start Airflow in the staging environment:

export AIRFLOW_HOME=/opt/airflow_staging
export AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://user:pass@staging-db:5432/airflow
airflow webserver

What will happen when you run this command?

medium

A. Airflow will start using the staging database and configs in /opt/airflow_staging

B. Airflow will start using the production database and default configs

C. Airflow will fail because environment variables cannot be set this way

D. Airflow will start but ignore the SQL connection string

Solution

1. Step 1: Analyze environment variables set before starting Airflow

   Setting AIRFLOW_HOME changes the config directory; setting AIRFLOW__CORE__SQL_ALCHEMY_CONN changes the database connection.

2. Step 2: Understand Airflow behavior on startup

   Airflow uses these environment variables to load configs and connect to the specified database.

3. Final Answer:

   Airflow will start using the staging database and configs in /opt/airflow_staging -> Option A

4. Quick Check:

   Env vars control config and DB connection [OK]

Hint: Env vars before airflow start set environment [OK]

Common Mistakes:

• Assuming Airflow ignores env vars
• Thinking Airflow uses production DB by default
• Believing env vars cannot be set inline

4. You deployed Airflow for prod but accidentally used the dev database connection string in the config. What is the best way to fix this?

medium

A. Change the airflow.cfg file in the dev environment

B. Delete all DAGs and restart Airflow

C. Run Airflow without any environment variables

D. Change the AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable to the production database and restart Airflow

Solution

1. Step 1: Identify the incorrect database connection setting

   The problem is the wrong database connection string used in production.

2. Step 2: Correct the environment variable and restart Airflow

   Updating AIRFLOW__CORE__SQL_ALCHEMY_CONN to the correct production DB and restarting applies the fix.

3. Final Answer:

   Change the AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable to the production database and restart Airflow -> Option D

4. Quick Check:

   Fix DB connection env var and restart [OK]

Hint: Fix DB env var, then restart Airflow [OK]

Common Mistakes:

• Deleting DAGs unnecessarily
• Ignoring environment variables
• Changing dev configs instead of prod

5. You want to deploy Airflow for dev, staging, and prod on the same server. Which approach ensures proper isolation and avoids conflicts?

hard

A. Run one Airflow instance and switch configs manually before each run

B. Run three separate Airflow instances with different AIRFLOW_HOME and database connections

C. Use the same database but different DAG folders for each environment

D. Run Airflow only in production mode and test DAGs there

Solution

1. Step 1: Understand isolation needs for multiple environments on one server

   Each environment needs its own config directory and database to avoid conflicts.

2. Step 2: Identify the correct deployment approach

   Running separate Airflow instances with distinct AIRFLOW_HOME and DB connections ensures isolation.

3. Final Answer:

   Run three separate Airflow instances with different AIRFLOW_HOME and database connections -> Option B

4. Quick Check:

   Separate instances + configs = isolation [OK]

Hint: Use separate AIRFLOW_HOME and DB per instance [OK]

Common Mistakes:

• Switching configs manually risks errors
• Sharing one database causes conflicts
• Running only production ignores testing needs