Leader and Follower in Kafka: Roles Explained Simply

In Kafka, a leader is the broker responsible for managing all read and write requests for a partition, while followers replicate the leader's data to stay in sync. This setup ensures data reliability and fault tolerance by allowing followers to take over if the leader fails.

How It Works

Imagine a classroom where one student is the leader who writes notes on the board, and other students are followers who copy those notes. In Kafka, each partition of a topic has one leader broker that handles all the data writes and reads. The followers are other brokers that keep copies of the leader's data by continuously copying what the leader writes.

This system helps keep data safe. If the leader broker crashes, one of the followers can quickly become the new leader, so the system keeps working without losing data. This way, Kafka balances workload and ensures no single point of failure.

Example

This example shows how to check the leader and followers of a Kafka topic partition using the Kafka command-line tool.

bash

kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Output

Topic: my-topic  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3

When to Use

Leader and follower roles are used automatically by Kafka to manage data replication and availability. You don't manually assign these roles, but understanding them helps when designing systems that need high availability and fault tolerance.

Use Kafka in scenarios where you need reliable message delivery, such as processing logs, tracking user activity, or building real-time data pipelines. The leader-follower setup ensures your data is safe even if some servers fail.

Key Points

• The leader handles all client requests for a partition.
• Followers replicate the leader's data to stay updated.
• If the leader fails, a follower is promoted to leader to keep the system running.
• This design provides fault tolerance and data reliability.

Key Takeaways

Kafka partitions have one leader broker that manages all reads and writes.

Followers replicate the leader's data to ensure fault tolerance.

If the leader fails, a follower automatically becomes the new leader.

This leader-follower model keeps Kafka highly available and reliable.

Understanding these roles helps in designing robust Kafka-based systems.