Multi-environment deployment (dev, staging, prod) in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of having multiple environments like dev, staging, and prod in Airflow deployment?

Multiple environments help separate development, testing, and production workflows. This prevents errors in development from affecting live data and allows safe testing before production deployment.

intermediate

How can Airflow configurations differ between dev, staging, and prod environments?

Configurations like database connections, executor types, and logging levels can be set differently for each environment to match their needs, such as using a LocalExecutor in dev and a CeleryExecutor in prod.

intermediate

What is the role of environment variables in managing Airflow multi-environment deployments?

Environment variables allow dynamic configuration of Airflow settings per environment without changing code. For example, setting AIRFLOW__CORE__SQL_ALCHEMY_CONN differently for dev, staging, and prod.

intermediate

Why is it important to use separate Airflow metadata databases for each environment?

Using separate metadata databases keeps workflow states isolated. This avoids conflicts and data mix-ups between dev, staging, and prod, ensuring accurate monitoring and execution.

advanced

Name a common strategy to deploy Airflow DAGs differently across environments.

A common strategy is to use separate DAG folders or branches in version control for each environment, or to use conditional logic in DAG code to behave differently based on environment variables.

Which Airflow component should be isolated between dev, staging, and prod environments?

A Metadata database

B Scheduler UI theme

C Task retry count

D Python version

The metadata database stores workflow states and must be separate to avoid conflicts between environments.

What is a recommended way to change Airflow configurations per environment?

A Use environment variables

B Restart Airflow without changes

C Change Airflow source code

D Hardcode values in DAG files

Environment variables allow flexible configuration without code changes.

In a multi-environment setup, which executor might be used in production for scalability?

A LocalExecutor

B CeleryExecutor

C SequentialExecutor

D DebugExecutor

CeleryExecutor supports distributed task execution, suitable for production.

Why use a staging environment in Airflow deployment?

A To run production workflows directly

B To develop new Airflow features

C To store logs permanently

D To test workflows before production

Staging is for testing workflows safely before deploying to production.

How can DAGs be managed differently across environments?

A By changing Airflow core code

B By disabling the scheduler

C Using separate DAG folders or branches

D By using the same DAGs everywhere without change

Separate folders or branches help isolate DAG versions per environment.

Explain how environment variables help manage Airflow configurations in dev, staging, and prod.

Think about how you can change settings without editing DAG files.

Describe the benefits of having separate metadata databases for each Airflow environment.

Consider what would happen if dev and prod shared the same database.