Leader election in Kafka - Step-by-Step Execution

Process Flow - Leader election

Start: Kafka Cluster

↓

Broker Startup

↓

Broker Registers with Zookeeper

↓

Zookeeper Elects Controller

↓

Controller Assigns Partition Leaders

↓

Brokers Accept Leader Roles

↓

Cluster Ready for Requests

Kafka brokers start and register with Zookeeper, which elects a controller broker. The controller assigns leaders for partitions, enabling the cluster to handle requests.

Execution Sample

Kafka

1. Broker starts and registers with Zookeeper
2. Zookeeper runs leader election
3. Controller broker is elected
4. Controller assigns partition leaders
5. Brokers accept leader roles

This sequence shows how Kafka brokers and Zookeeper interact to elect a controller and assign leaders for partitions.

Process Table

Step	Action	Component	Result	Next Step
1	Broker starts	Broker 1	Broker 1 online	Register with Zookeeper
2	Broker registers	Broker 1	Broker 1 registered	Wait for election
3	Leader election starts	Zookeeper	Election in progress	Select controller
4	Controller elected	Broker 1	Broker 1 is controller	Assign partition leaders
5	Assign leaders	Controller	Partition 0 leader: Broker 1	Notify brokers
6	Brokers accept roles	Broker 1	Broker 1 leader for partition 0	Cluster ready
7	Cluster ready	All brokers	Leaders assigned, ready for requests	End

💡 All brokers have leaders assigned; cluster is ready to serve requests.

Status Tracker

Variable	Start	After Step 2	After Step 4	After Step 6	Final
Broker 1 status	offline	registered	controller	leader for partition 0	ready
Zookeeper election	none	in progress	controller selected	leaders assigned	complete

Key Moments - 3 Insights

Why does Zookeeper elect a controller broker?

Zookeeper elects a single controller to coordinate leader assignments and manage cluster state, ensuring only one broker controls partition leadership at a time (see execution_table step 4).

What happens if the controller broker fails?

Zookeeper detects the failure and triggers a new leader election to select a new controller, ensuring continuous cluster management (related to execution_table step 3).

Why must brokers accept leader roles after assignment?

Brokers must acknowledge their leader roles to start handling client requests for their partitions, confirming cluster readiness (see execution_table step 6).

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the controller broker elected?

A Step 4

B Step 3

C Step 5

D Step 6

💡 Hint

Check the 'Result' column for 'Broker 1 is controller' in execution_table.

According to variable_tracker, what is Broker 1's status after step 6?

A offline

B registered

C leader for partition 0

D controller

💡 Hint

Look at the 'Broker 1 status' row under 'After Step 6' in variable_tracker.

If Broker 1 fails after being elected controller, what happens next according to the concept flow?

A Cluster stops working permanently

B Zookeeper triggers a new leader election

C Broker 1 remains controller

D Partition leaders are not assigned

💡 Hint

Refer to key_moments about controller failure and execution_table step 3.

Concept Snapshot

Kafka leader election:
- Brokers register with Zookeeper
- Zookeeper elects one controller broker
- Controller assigns partition leaders
- Brokers accept leader roles
- Ensures cluster coordination and availability

Full Transcript

In Kafka, leader election is the process where brokers start and register with Zookeeper. Zookeeper then elects one broker as the controller. This controller assigns leaders for each partition. Brokers accept these leader roles to handle client requests. If the controller fails, Zookeeper triggers a new election to keep the cluster running smoothly.