High availability configuration in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is high availability (HA) in Airflow?

High availability in Airflow means setting up the system so it keeps running without interruption, even if some parts fail. It ensures workflows continue without downtime.

intermediate

Name two key components to configure for Airflow high availability.

1. Multiple Airflow schedulers running in parallel. 2. A highly available metadata database (like PostgreSQL with replication).

intermediate

Why use a message broker like Redis or RabbitMQ in Airflow HA setup?

A message broker helps coordinate tasks between multiple schedulers and workers, ensuring task distribution and avoiding conflicts.

beginner

What role does the metadata database play in Airflow high availability?

The metadata database stores the state of all workflows and tasks. Making it highly available prevents data loss and keeps Airflow running smoothly.

beginner

How does running multiple schedulers improve Airflow availability?

Multiple schedulers share the workload and take over if one fails, so workflows keep running without interruption.

Which component is essential for Airflow high availability to avoid a single point of failure?

A Single worker node

B Single scheduler instance

C Local file storage

D Metadata database with replication

A replicated metadata database prevents data loss and downtime if one database instance fails.

What is the purpose of running multiple Airflow schedulers in HA setup?

A To balance task scheduling and provide failover

B To reduce the number of workers needed

C To store logs more efficiently

D To increase the number of DAGs

Multiple schedulers share the scheduling load and provide backup if one scheduler fails.

Which message broker is commonly used in Airflow HA for task coordination?

A MySQL

B Redis

C MongoDB

D SQLite

Redis is a popular message broker used to coordinate tasks between schedulers and workers.

What happens if the Airflow metadata database is not highly available?

A DAGs will execute twice

B Schedulers will run faster

C Airflow may lose task state and stop working properly

D Workers will automatically restart

Without a highly available database, Airflow can lose critical state information causing failures.

Which of these is NOT a benefit of Airflow high availability?

A Elimination of all bugs in DAGs

B Automatic failover of schedulers

C Zero downtime for workflows

D Guaranteed task execution order without conflicts

High availability improves uptime and failover but does not fix bugs in workflow code.

Explain how multiple schedulers and a replicated metadata database work together to provide high availability in Airflow.

Think about how tasks keep running even if one part stops working.

Describe the role of a message broker in an Airflow high availability setup.

It's like a traffic controller for tasks.

Practice

1. What is the main purpose of configuring high availability in Apache Airflow?

easy

A. To reduce the number of DAGs in the system

B. To keep Airflow running even if some components fail

C. To speed up the Airflow UI loading time

D. To limit the number of users accessing Airflow

Solution

1. Step 1: Understand high availability concept

   High availability means the system keeps working even if parts fail.

2. Step 2: Apply to Airflow context

   In Airflow, this means tasks and services continue running despite failures.

3. Final Answer:

   To keep Airflow running even if some components fail -> Option B

4. Quick Check:

   High availability = system stays up despite failures [OK]

Hint: High availability means no downtime during failures [OK]

Common Mistakes:

• Confusing high availability with performance optimization
• Thinking it limits users or DAGs
• Assuming it only affects the UI

2. Which executor is commonly used in Airflow for high availability setups?

easy

A. SequentialExecutor

B. DebugExecutor

C. LocalExecutor

D. CeleryExecutor

Solution

1. Step 1: Recall executor types in Airflow

   SequentialExecutor runs tasks one by one, LocalExecutor runs tasks locally in parallel.

2. Step 2: Identify executor for distributed task handling

   CeleryExecutor uses workers and message brokers for distributed task execution, suitable for high availability.

3. Final Answer:

   CeleryExecutor -> Option D

4. Quick Check:

   Distributed task handling = CeleryExecutor [OK]

Hint: CeleryExecutor enables distributed task execution [OK]

Common Mistakes:

• Choosing SequentialExecutor which is single-threaded
• Confusing LocalExecutor with distributed execution
• Selecting DebugExecutor which is for testing only

3. Given this Airflow configuration snippet for high availability, what is the role of Redis?

[celery]
broker_url = redis://localhost:6379/0

medium

A. Stores task results permanently

B. Manages the Airflow metadata database

C. Acts as a message broker between scheduler and workers

D. Runs the Airflow webserver

Solution

1. Step 1: Identify Redis role in CeleryExecutor

   Redis is used as a message broker to pass task messages between scheduler and workers.

2. Step 2: Differentiate Redis from other components

   Redis is not for storing results permanently or running webserver or metadata DB.

3. Final Answer:

   Acts as a message broker between scheduler and workers -> Option C

4. Quick Check:

   Redis = message broker in CeleryExecutor [OK]

Hint: Redis brokers messages between scheduler and workers [OK]

Common Mistakes:

• Thinking Redis stores metadata or results
• Confusing Redis with PostgreSQL database
• Assuming Redis runs the webserver

4. You configured Airflow with CeleryExecutor but tasks are not being distributed to workers. Which configuration mistake could cause this?

medium

A. Using SQLite as the metadata database

B. Setting broker_url to a Redis server

C. Running multiple scheduler instances

D. Enabling high availability mode in webserver

Solution

1. Step 1: Understand metadata database requirements

   CeleryExecutor requires a robust metadata database like PostgreSQL, not SQLite.

2. Step 2: Identify why SQLite causes issues

   SQLite locks the database and does not support concurrent writes well, blocking task distribution.

3. Final Answer:

   Using SQLite as the metadata database -> Option A

4. Quick Check:

   SQLite blocks concurrency, breaks CeleryExecutor [OK]

Hint: Use PostgreSQL, not SQLite, for CeleryExecutor metadata DB [OK]

Common Mistakes:

• Assuming Redis broker_url is wrong when it is correct
• Thinking multiple schedulers cause task distribution failure
• Confusing webserver settings with task distribution

5. To achieve high availability for Airflow's scheduler, which setup is recommended?

hard

A. Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor

B. Run a single scheduler instance with SQLite backend

C. Run multiple webservers without a scheduler

D. Use LocalExecutor with a single scheduler instance

Solution

1. Step 1: Understand scheduler high availability needs

   Multiple scheduler instances increase reliability and avoid single point of failure.

2. Step 2: Combine with proper backend and executor

   PostgreSQL supports concurrency well; CeleryExecutor enables distributed task execution.

3. Step 3: Eliminate incorrect options

   SQLite and single scheduler limit availability; webservers alone don't schedule tasks; LocalExecutor is not distributed.

4. Final Answer:

   Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor -> Option A

5. Quick Check:

   Multiple schedulers + PostgreSQL + CeleryExecutor = HA scheduler [OK]

Hint: Multiple schedulers + PostgreSQL + CeleryExecutor = high availability [OK]

Common Mistakes:

• Using SQLite which blocks concurrency
• Running only one scheduler instance
• Confusing webserver role with scheduler