Multi-environment deployment (dev, staging, prod) in Apache Airflow - Step-by-Step Execution

Process Flow - Multi-environment deployment (dev, staging, prod)

Write DAG code

↓

Set environment variables

↓

Configure airflow.cfg or use variables

↓

Deploy DAG to environment folder

↓

Airflow scheduler picks DAG

↓

Run DAG with environment-specific configs

↓

Monitor logs and outputs

↓

Repeat for dev, staging, prod

This flow shows how the same Airflow DAG code is deployed to different environments with specific settings, allowing safe testing and production runs.

Execution Sample

Apache Airflow

import os
from airflow import DAG
from airflow.operators.python import PythonOperator

def print_env():
    print(f"Running in {os.getenv('ENV')} environment")

This DAG prints the current environment by reading the ENV variable, showing how environment affects execution.

Process Table

Step	Action	Environment Variable ENV	DAG Behavior	Output
1	Set ENV=dev	dev	DAG reads ENV=dev	Prints: Running in dev environment
2	Deploy DAG to dev folder	dev	Scheduler picks DAG	DAG ready in dev
3	Run DAG in dev	dev	print_env runs	Running in dev environment
4	Set ENV=staging	staging	DAG reads ENV=staging	Prints: Running in staging environment
5	Deploy DAG to staging folder	staging	Scheduler picks DAG	DAG ready in staging
6	Run DAG in staging	staging	print_env runs	Running in staging environment
7	Set ENV=prod	prod	DAG reads ENV=prod	Prints: Running in prod environment
8	Deploy DAG to prod folder	prod	Scheduler picks DAG	DAG ready in prod
9	Run DAG in prod	prod	print_env runs	Running in prod environment
10	End	-	-	All environments deployed and tested

💡 All environments have been deployed and DAGs executed with their specific ENV variable.

Status Tracker

Variable	Start	After Step 1	After Step 4	After Step 7	Final
ENV	undefined	dev	staging	prod	prod

Key Moments - 3 Insights

Why does the DAG print different environment names even though the code is the same?

Because the ENV variable changes before deployment (see execution_table steps 1,4,7), the DAG reads this variable at runtime and behaves accordingly.

What happens if you forget to set the ENV variable before deploying?

The DAG will print 'Running in None environment' or error because ENV is not set, as shown by the variable_tracker start value.

Why do we deploy the same DAG code to different folders?

To isolate environments so changes in dev or staging don't affect prod, as shown in execution_table steps 2,5,8.

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of ENV when the DAG runs in staging?

A staging

B dev

C prod

D undefined

💡 Hint

Check the 'Environment Variable ENV' column at step 6 where DAG runs in staging.

At which step does the DAG first print 'Running in prod environment'?

A Step 7

B Step 8

C Step 9

D Step 10

💡 Hint

Look at the 'Output' column for the prod environment print statement.

If ENV was never set, what would the DAG output be when run?

A Running in dev environment

B Running in None environment

C Error: ENV not found

D Running in prod environment

💡 Hint

Refer to variable_tracker start value and how print_env uses os.getenv('ENV').

Concept Snapshot

Multi-environment deployment in Airflow:
- Use same DAG code for dev, staging, prod
- Set ENV variable differently per environment
- Deploy DAGs to separate folders
- DAG reads ENV at runtime to adjust behavior
- Allows safe testing before production run

Full Transcript

This visual execution shows how to deploy the same Airflow DAG code to multiple environments: development, staging, and production. We set an environment variable called ENV to identify the environment. The DAG reads this variable at runtime and prints which environment it is running in. We deploy the DAG to separate folders for each environment so they do not interfere. The scheduler picks up the DAG in each environment and runs it with the correct settings. This approach helps test changes safely in dev and staging before deploying to production.