High availability configuration in Apache Airflow - Step-by-Step Execution

Process Flow - High availability configuration

Start Airflow Scheduler

↓

Check if Leader Election Enabled?

No→Multiple Schedulers Active

Yes↓

Participate in Leader Election

↓

Is this Scheduler the Leader?

No→Wait and Monitor

Yes↓

Run Scheduler Tasks

↓

Monitor Health and Failover

↓

If Leader Fails, New Leader Elected

↓

Continue Scheduling with New Leader

This flow shows how Airflow schedulers coordinate to ensure only one active leader schedules tasks, enabling high availability.

Execution Sample

Apache Airflow

AIRFLOW__scheduler__enable_leader_election=true
airflow scheduler
# Multiple schedulers start
# Leader election happens
# Only leader schedules DAG runs
# Others standby and monitor

This code starts Airflow schedulers with leader election enabled to ensure high availability.

Process Table

Step	Action	Leader Election Status	Scheduler Role	Result
1	Start multiple schedulers	Enabled	None yet	Schedulers start and prepare for leader election
2	Leader election begins	Enabled	Election in progress	Schedulers compete to become leader
3	Scheduler A wins election	Enabled	Leader	Scheduler A starts scheduling tasks
4	Schedulers B and C lose election	Enabled	Standby	Schedulers wait and monitor leader health
5	Leader Scheduler A fails	Enabled	Leader down	Leader election restarts
6	Scheduler B wins new election	Enabled	Leader	Scheduler B takes over scheduling
7	Schedulers A and C standby	Enabled	Standby	Schedulers monitor new leader
8	Leader Scheduler B continues scheduling	Enabled	Leader	High availability maintained

💡 Leader scheduler continues running; standby schedulers monitor and wait to take over if leader fails

Status Tracker

Variable	Start	After Step 3	After Step 5	After Step 6	Final
Scheduler A Role	None	Leader	Down	Down	Down
Scheduler B Role	None	Standby	Standby	Leader	Leader
Scheduler C Role	None	Standby	Standby	Standby	Standby
Leader Election Status	Enabled	Enabled	Enabled	Enabled	Enabled

Key Moments - 3 Insights

Why does only one scheduler run tasks even though multiple schedulers start?

Because leader election ensures only one scheduler is the leader at a time (see execution_table step 3), and only the leader schedules tasks while others wait.

What happens when the leader scheduler fails?

Leader election restarts (execution_table step 5), and a new scheduler becomes leader to continue scheduling, ensuring no downtime.

Do standby schedulers do nothing while waiting?

No, they monitor the leader's health and are ready to take over immediately if the leader fails (execution_table steps 4 and 7).

Visual Quiz - 3 Questions

Look at the execution table, which scheduler becomes leader after the first election?

A Scheduler C

B Scheduler B

C Scheduler A

D No scheduler becomes leader

💡 Hint

Check execution_table row 3 under 'Scheduler Role'

At which step does the leader scheduler fail?

A Step 5

B Step 3

C Step 6

D Step 8

💡 Hint

Look for 'Leader down' status in execution_table

If leader election was disabled, what would happen?

A No scheduler runs tasks

B Multiple schedulers run tasks simultaneously

C Only one scheduler runs tasks

D Schedulers wait for leader election

💡 Hint

Refer to concept_flow where 'No' branch leads to multiple schedulers active without election

Concept Snapshot

Airflow High Availability uses leader election among schedulers.
Only one scheduler acts as leader to run tasks.
Others standby and monitor leader health.
If leader fails, a new leader is elected automatically.
This prevents downtime and task duplication.

Full Transcript

High availability in Airflow means running multiple schedulers but only one schedules tasks at a time. This is done by enabling leader election. When schedulers start, they compete to become the leader. The leader runs the scheduling tasks. The others wait and watch the leader's health. If the leader fails, a new leader is chosen automatically. This keeps Airflow running without interruption.