Leader election in Kafka - Commands & Configuration

Introduction

Leader election is the process where Kafka brokers decide which broker will manage a partition's operations. This ensures only one broker coordinates reads and writes for a partition, preventing conflicts and data loss.

When a Kafka broker fails and another broker must take over as leader for its partitions.

When adding new brokers to a Kafka cluster and leaders need to be balanced across them.

When manually triggering leader re-election to improve cluster performance or fix stuck leaders.

When monitoring cluster health and verifying leader assignments for partitions.

When performing maintenance on brokers and needing to move leadership away from them.

Commands

This command shows the current leader and replicas for each partition of 'my-topic'. It helps verify which broker is leader before and after leader election.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-topic

Expected Output

Topic: my-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: 

	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: my-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: my-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--bootstrap-server - Specifies the Kafka broker to connect to

→

--describe - Shows detailed information about the topic partitions

→

--topic - Specifies the topic to describe

This command triggers a preferred leader election, moving leadership to the preferred replicas for partitions. Useful to rebalance leaders after broker recovery or maintenance.

Terminal

kafka-preferred-replica-election.sh --bootstrap-server localhost:9092

Expected Output

Starting preferred replica leader election for partitions in all topics.

Preferred replica leader election completed successfully.

→

--bootstrap-server - Specifies the Kafka broker to connect to

Run again to verify that the leader for each partition has changed to the preferred replica after the election.

Terminal

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic my-topic

Expected Output

Topic: my-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: 

	Topic: my-topic	Partition: 0	Leader: 2	Replicas: 1,2	Isr: 1,2

	Topic: my-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: my-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--bootstrap-server - Specifies the Kafka broker to connect to

→

--describe - Shows detailed information about the topic partitions

→

--topic - Specifies the topic to describe

Key Concept

If you remember nothing else from leader election, remember: only one broker leads a partition at a time to keep data consistent and available.

Common Mistakes

Running preferred replica election without checking current leaders

You might trigger unnecessary leader changes causing temporary unavailability or load spikes.

Always describe topic partitions first to understand current leader distribution before triggering election.

Not specifying the correct bootstrap server address

The command will fail to connect or affect the wrong Kafka cluster.

Use the correct --bootstrap-server address matching your Kafka cluster.

Expecting leader election to fix all partition issues automatically

Leader election only changes leadership; it does not fix underlying broker or network problems.

Investigate broker health and network status if leader election does not resolve issues.

Summary

Use kafka-topics.sh --describe to check current partition leaders.

Trigger leader election with kafka-preferred-replica-election.sh to move leadership to preferred brokers.

Verify leader changes by describing the topic partitions again.