Leader election in Kafka - Step-by-Step Execution

Process Flow - Leader election

Start: Kafka Cluster

↓

Broker Startup

↓

Broker Registers with Zookeeper

↓

Zookeeper Elects Controller

↓

Controller Assigns Partition Leaders

↓

Brokers Accept Leader Roles

↓

Cluster Ready for Requests

Kafka brokers start and register with Zookeeper, which elects a controller broker. The controller assigns leaders for partitions, enabling the cluster to handle requests.

Execution Sample

Kafka

1. Broker starts and registers with Zookeeper
2. Zookeeper runs leader election
3. Controller broker is elected
4. Controller assigns partition leaders
5. Brokers accept leader roles

This sequence shows how Kafka brokers and Zookeeper interact to elect a controller and assign leaders for partitions.

Process Table

Step	Action	Component	Result	Next Step
1	Broker starts	Broker 1	Broker 1 online	Register with Zookeeper
2	Broker registers	Broker 1	Broker 1 registered	Wait for election
3	Leader election starts	Zookeeper	Election in progress	Select controller
4	Controller elected	Broker 1	Broker 1 is controller	Assign partition leaders
5	Assign leaders	Controller	Partition 0 leader: Broker 1	Notify brokers
6	Brokers accept roles	Broker 1	Broker 1 leader for partition 0	Cluster ready
7	Cluster ready	All brokers	Leaders assigned, ready for requests	End

💡 All brokers have leaders assigned; cluster is ready to serve requests.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
Broker 1 status	offline	registered	controller	leader for partition 0	ready
Zookeeper election	none	in progress	controller selected	leaders assigned	complete

Key Moments - 3 Insights

Why does Zookeeper elect a controller broker?

Zookeeper elects a single controller to coordinate leader assignments and manage cluster state, ensuring only one broker controls partition leadership at a time (see execution_table step 4).

What happens if the controller broker fails?

Zookeeper detects the failure and triggers a new leader election to select a new controller, ensuring continuous cluster management (related to execution_table step 3).

Why must brokers accept leader roles after assignment?

Brokers must acknowledge their leader roles to start handling client requests for their partitions, confirming cluster readiness (see execution_table step 6).

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the controller broker elected?

A Step 4

B Step 3

C Step 5

D Step 6

💡 Hint

Check the 'Result' column for 'Broker 1 is controller' in execution_table.

According to variable_tracker, what is Broker 1's status after step 6?

A offline

B registered

C leader for partition 0

D controller

💡 Hint

Look at the 'Broker 1 status' row under 'After Step 6' in variable_tracker.

If Broker 1 fails after being elected controller, what happens next according to the concept flow?

A Cluster stops working permanently

B Zookeeper triggers a new leader election

C Broker 1 remains controller

D Partition leaders are not assigned

💡 Hint

Refer to key_moments about controller failure and execution_table step 3.

Concept Snapshot

Kafka leader election:
- Brokers register with Zookeeper
- Zookeeper elects one controller broker
- Controller assigns partition leaders
- Brokers accept leader roles
- Ensures cluster coordination and availability

Full Transcript

In Kafka, leader election is the process where brokers start and register with Zookeeper. Zookeeper then elects one broker as the controller. This controller assigns leaders for each partition. Brokers accept these leader roles to handle client requests. If the controller fails, Zookeeper triggers a new election to keep the cluster running smoothly.

Practice

1. What is the main purpose of leader election in Kafka?

easy

A. To backup data to external storage

B. To balance the load evenly among all brokers

C. To choose one broker to lead and coordinate tasks

D. To monitor client connections

Solution

1. Step 1: Understand leader election role

   Leader election picks one broker as the leader to manage partition tasks.

2. Step 2: Compare options

   Only To choose one broker to lead and coordinate tasks describes choosing a broker to lead and coordinate tasks, which matches Kafka's leader election purpose.

3. Final Answer:

   To choose one broker to lead and coordinate tasks -> Option C

4. Quick Check:

   Leader election = choose leader broker [OK]

Hint: Leader election picks one leader broker, not all brokers [OK]

Common Mistakes:

• Confusing leader election with load balancing
• Thinking leader election backs up data
• Assuming leader election monitors clients

2. Which of the following is the correct way to configure Kafka broker IDs for leader election?

easy

A. broker.id=unique_integer_per_broker

B. broker.id=localhost

C. broker.id=auto_assign

D. broker.id=0.0.0.0

Solution

1. Step 1: Identify broker ID type

   Kafka requires each broker to have a unique integer ID for leader election.

2. Step 2: Check options

   Only broker.id=unique_integer_per_broker uses a unique integer, while others use invalid or non-integer values.

3. Final Answer:

   broker.id=unique_integer_per_broker -> Option A

4. Quick Check:

   Broker ID must be unique integer [OK]

Hint: Broker ID must be a unique integer, not a string or IP [OK]

Common Mistakes:

• Using hostnames or IPs as broker IDs
• Setting broker.id to zero or non-unique values
• Assuming auto assignment works without config

3. Given the following Kafka cluster state, which broker will be the leader for partition 0?

Broker 1: ISR = [1, 2, 3]
Broker 2: ISR = [1, 2, 3]
Broker 3: ISR = [1, 2, 3]
Current leader: Broker 2
Broker 2 fails

medium

A. Broker 2

B. Broker 1

C. Broker 3

D. No leader assigned

Solution

1. Step 1: Understand ISR and leader failover

   ISR (In-Sync Replicas) are eligible leaders. When current leader fails, next ISR broker becomes leader.

2. Step 2: Identify next leader after Broker 2 fails

   Broker 1 and 3 are in ISR. Kafka picks the lowest broker ID in ISR as new leader, so Broker 1.

3. Final Answer:

   Broker 1 -> Option B

4. Quick Check:

   Next leader = lowest ISR broker after failure [OK]

Hint: Leader election picks lowest ISR broker if current leader fails [OK]

Common Mistakes:

• Choosing failed broker as leader
• Picking broker outside ISR
• Assuming no leader assigned after failure

4. You have a Kafka cluster where leader election is not working properly. Which of the following is the most likely cause?

medium

A. Each broker has a unique broker.id

B. Zookeeper is running and connected

C. Partitions have multiple replicas

D. All brokers have the same broker.id

Solution

1. Step 1: Identify leader election requirements

   Leader election requires each broker to have a unique broker.id to distinguish them.

2. Step 2: Analyze causes

   If all brokers share the same broker.id, leader election cannot assign a unique leader, causing failure.

3. Final Answer:

   All brokers have the same broker.id -> Option D

4. Quick Check:

   Unique broker IDs needed for leader election [OK]

Hint: Check broker IDs; duplicates break leader election [OK]

Common Mistakes:

• Assuming Zookeeper connection causes failure
• Thinking multiple replicas cause election failure
• Ignoring broker.id uniqueness

5. In a Kafka cluster using KRaft mode (without Zookeeper), how does leader election differ from the traditional Zookeeper-based approach?

hard

A. KRaft uses an internal quorum of controllers to elect leaders without external Zookeeper

B. KRaft requires manual leader assignment by the admin

C. KRaft disables leader election and uses round-robin partition assignment

D. KRaft uses Zookeeper but with a different election algorithm

Solution

1. Step 1: Understand KRaft mode leader election

   KRaft mode replaces Zookeeper with an internal controller quorum that manages leader election.

2. Step 2: Compare options

   Only KRaft uses an internal quorum of controllers to elect leaders without external Zookeeper correctly states KRaft uses internal quorum without external Zookeeper for leader election.

3. Final Answer:

   KRaft uses an internal quorum of controllers to elect leaders without external Zookeeper -> Option A

4. Quick Check:

   KRaft = internal quorum leader election [OK]

Hint: KRaft replaces Zookeeper with internal controller quorum [OK]

Common Mistakes:

• Thinking KRaft disables leader election
• Assuming manual leader assignment in KRaft
• Believing KRaft still uses external Zookeeper