High availability configuration in Apache Airflow - Step-by-Step Execution

Process Flow - High availability configuration

Start Airflow Scheduler

↓

Check if Leader Election Enabled?

No→Multiple Schedulers Active

Yes↓

Participate in Leader Election

↓

Is this Scheduler the Leader?

No→Wait and Monitor

Yes↓

Run Scheduler Tasks

↓

Monitor Health and Failover

↓

If Leader Fails, New Leader Elected

↓

Continue Scheduling with New Leader

This flow shows how Airflow schedulers coordinate to ensure only one active leader schedules tasks, enabling high availability.

Execution Sample

Apache Airflow

AIRFLOW__scheduler__enable_leader_election=true
airflow scheduler
# Multiple schedulers start
# Leader election happens
# Only leader schedules DAG runs
# Others standby and monitor

This code starts Airflow schedulers with leader election enabled to ensure high availability.

Process Table

Step	Action	Leader Election Status	Scheduler Role	Result
1	Start multiple schedulers	Enabled	None yet	Schedulers start and prepare for leader election
2	Leader election begins	Enabled	Election in progress	Schedulers compete to become leader
3	Scheduler A wins election	Enabled	Leader	Scheduler A starts scheduling tasks
4	Schedulers B and C lose election	Enabled	Standby	Schedulers wait and monitor leader health
5	Leader Scheduler A fails	Enabled	Leader down	Leader election restarts
6	Scheduler B wins new election	Enabled	Leader	Scheduler B takes over scheduling
7	Schedulers A and C standby	Enabled	Standby	Schedulers monitor new leader
8	Leader Scheduler B continues scheduling	Enabled	Leader	High availability maintained

💡 Leader scheduler continues running; standby schedulers monitor and wait to take over if leader fails

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 6	Final
Scheduler A Role	None	Leader	Down	Down	Down
Scheduler B Role	None	Standby	Standby	Leader	Leader
Scheduler C Role	None	Standby	Standby	Standby	Standby
Leader Election Status	Enabled	Enabled	Enabled	Enabled	Enabled

Key Moments - 3 Insights

Why does only one scheduler run tasks even though multiple schedulers start?

Because leader election ensures only one scheduler is the leader at a time (see execution_table step 3), and only the leader schedules tasks while others wait.

What happens when the leader scheduler fails?

Leader election restarts (execution_table step 5), and a new scheduler becomes leader to continue scheduling, ensuring no downtime.

Do standby schedulers do nothing while waiting?

No, they monitor the leader's health and are ready to take over immediately if the leader fails (execution_table steps 4 and 7).

Visual Quiz - 3 Questions

Look at the execution table, which scheduler becomes leader after the first election?

A Scheduler C

B Scheduler B

C Scheduler A

D No scheduler becomes leader

💡 Hint

Check execution_table row 3 under 'Scheduler Role'

At which step does the leader scheduler fail?

A Step 5

B Step 3

C Step 6

D Step 8

💡 Hint

Look for 'Leader down' status in execution_table

If leader election was disabled, what would happen?

A No scheduler runs tasks

B Multiple schedulers run tasks simultaneously

C Only one scheduler runs tasks

D Schedulers wait for leader election

💡 Hint

Refer to concept_flow where 'No' branch leads to multiple schedulers active without election

Concept Snapshot

Airflow High Availability uses leader election among schedulers.
Only one scheduler acts as leader to run tasks.
Others standby and monitor leader health.
If leader fails, a new leader is elected automatically.
This prevents downtime and task duplication.

Full Transcript

High availability in Airflow means running multiple schedulers but only one schedules tasks at a time. This is done by enabling leader election. When schedulers start, they compete to become the leader. The leader runs the scheduling tasks. The others wait and watch the leader's health. If the leader fails, a new leader is chosen automatically. This keeps Airflow running without interruption.

Practice

1. What is the main purpose of configuring high availability in Apache Airflow?

easy

A. To reduce the number of DAGs in the system

B. To keep Airflow running even if some components fail

C. To speed up the Airflow UI loading time

D. To limit the number of users accessing Airflow

Solution

1. Step 1: Understand high availability concept

   High availability means the system keeps working even if parts fail.

2. Step 2: Apply to Airflow context

   In Airflow, this means tasks and services continue running despite failures.

3. Final Answer:

   To keep Airflow running even if some components fail -> Option B

4. Quick Check:

   High availability = system stays up despite failures [OK]

Hint: High availability means no downtime during failures [OK]

Common Mistakes:

• Confusing high availability with performance optimization
• Thinking it limits users or DAGs
• Assuming it only affects the UI

2. Which executor is commonly used in Airflow for high availability setups?

easy

A. SequentialExecutor

B. DebugExecutor

C. LocalExecutor

D. CeleryExecutor

Solution

1. Step 1: Recall executor types in Airflow

   SequentialExecutor runs tasks one by one, LocalExecutor runs tasks locally in parallel.

2. Step 2: Identify executor for distributed task handling

   CeleryExecutor uses workers and message brokers for distributed task execution, suitable for high availability.

3. Final Answer:

   CeleryExecutor -> Option D

4. Quick Check:

   Distributed task handling = CeleryExecutor [OK]

Hint: CeleryExecutor enables distributed task execution [OK]

Common Mistakes:

• Choosing SequentialExecutor which is single-threaded
• Confusing LocalExecutor with distributed execution
• Selecting DebugExecutor which is for testing only

3. Given this Airflow configuration snippet for high availability, what is the role of Redis?

[celery]
broker_url = redis://localhost:6379/0

medium

A. Stores task results permanently

B. Manages the Airflow metadata database

C. Acts as a message broker between scheduler and workers

D. Runs the Airflow webserver

Solution

1. Step 1: Identify Redis role in CeleryExecutor

   Redis is used as a message broker to pass task messages between scheduler and workers.

2. Step 2: Differentiate Redis from other components

   Redis is not for storing results permanently or running webserver or metadata DB.

3. Final Answer:

   Acts as a message broker between scheduler and workers -> Option C

4. Quick Check:

   Redis = message broker in CeleryExecutor [OK]

Hint: Redis brokers messages between scheduler and workers [OK]

Common Mistakes:

• Thinking Redis stores metadata or results
• Confusing Redis with PostgreSQL database
• Assuming Redis runs the webserver

4. You configured Airflow with CeleryExecutor but tasks are not being distributed to workers. Which configuration mistake could cause this?

medium

A. Using SQLite as the metadata database

B. Setting broker_url to a Redis server

C. Running multiple scheduler instances

D. Enabling high availability mode in webserver

Solution

1. Step 1: Understand metadata database requirements

   CeleryExecutor requires a robust metadata database like PostgreSQL, not SQLite.

2. Step 2: Identify why SQLite causes issues

   SQLite locks the database and does not support concurrent writes well, blocking task distribution.

3. Final Answer:

   Using SQLite as the metadata database -> Option A

4. Quick Check:

   SQLite blocks concurrency, breaks CeleryExecutor [OK]

Hint: Use PostgreSQL, not SQLite, for CeleryExecutor metadata DB [OK]

Common Mistakes:

• Assuming Redis broker_url is wrong when it is correct
• Thinking multiple schedulers cause task distribution failure
• Confusing webserver settings with task distribution

5. To achieve high availability for Airflow's scheduler, which setup is recommended?

hard

A. Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor

B. Run a single scheduler instance with SQLite backend

C. Run multiple webservers without a scheduler

D. Use LocalExecutor with a single scheduler instance

Solution

1. Step 1: Understand scheduler high availability needs

   Multiple scheduler instances increase reliability and avoid single point of failure.

2. Step 2: Combine with proper backend and executor

   PostgreSQL supports concurrency well; CeleryExecutor enables distributed task execution.

3. Step 3: Eliminate incorrect options

   SQLite and single scheduler limit availability; webservers alone don't schedule tasks; LocalExecutor is not distributed.

4. Final Answer:

   Run multiple scheduler instances with PostgreSQL backend and CeleryExecutor -> Option A

5. Quick Check:

   Multiple schedulers + PostgreSQL + CeleryExecutor = HA scheduler [OK]

Hint: Multiple schedulers + PostgreSQL + CeleryExecutor = high availability [OK]

Common Mistakes:

• Using SQLite which blocks concurrency
• Running only one scheduler instance
• Confusing webserver role with scheduler