Replication factor in Kafka - Step-by-Step Execution

Process Flow - Replication factor

Create Topic Request

↓

Assign Replicas per Partition

↓

Distribute Replicas Across Brokers

↓

Start Replication

↓

Leader Election for Each Partition

↓

Clients Produce/Consume Data

↓

Replication Maintains Data Copies

When a topic is created, Kafka assigns multiple copies (replicas) of each partition across brokers to ensure data safety and availability.

Execution Sample

Kafka

kafka-topics.sh --create --topic my-topic --partitions 2 --replication-factor 3 --bootstrap-server localhost:9092

This command creates a topic named 'my-topic' with 2 partitions and 3 replicas for each partition.

Process Table

Step	Action	Partition	Replicas Assigned	Leader Broker	Notes
1	Create topic request received	-	-	-	Topic 'my-topic' creation starts
2	Assign replicas	Partition 0	Broker 1, Broker 2, Broker 3	-	Replicas spread across brokers
3	Assign replicas	Partition 1	Broker 2, Broker 3, Broker 1	-	Replicas assigned for partition 1
4	Leader election	Partition 0	Broker 1, Broker 2, Broker 3	Broker 1	Leader chosen for partition 0
5	Leader election	Partition 1	Broker 2, Broker 3, Broker 1	Broker 2	Leader chosen for partition 1
6	Start replication	Partition 0	Broker 1, Broker 2, Broker 3	Broker 1	Leader replicates data to followers
7	Start replication	Partition 1	Broker 2, Broker 3, Broker 1	Broker 2	Leader replicates data to followers
8	Clients produce/consume	-	-	-	Data is safely stored with replication
9	Exit	-	-	-	Topic created with replication factor 3

💡 All partitions assigned 3 replicas; leaders elected; replication started successfully.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	Final
Partition 0 Replicas	None	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3
Partition 1 Replicas	None	None	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1
Partition 0 Leader	None	None	None	Broker 1	Broker 1	Broker 1	Broker 1	Broker 1
Partition 1 Leader	None	None	None	None	Broker 2	Broker 2	Broker 2	Broker 2

Key Moments - 3 Insights

Why does each partition have multiple replicas?

Each partition has multiple replicas to keep copies of data on different brokers. This protects data if one broker fails, as shown in steps 2 and 3 where replicas are assigned.

What is the role of the leader broker for a partition?

The leader broker handles all read and write requests for its partition and replicates data to follower brokers. This is clear in steps 6 and 7 where leaders start replication.

Can replication factor be larger than the number of brokers?

No, replication factor cannot exceed the number of brokers because each replica must be on a different broker. This is why in step 2 replicas are assigned to distinct brokers.

Visual Quiz - 3 Questions

Look at the execution table, which broker is the leader for Partition 1 after assignment?

A No leader assigned yet

B Broker 1

C Broker 3

D Broker 2

💡 Hint

Check the 'Leader Broker' column for Partition 1 in step 3.

At which step does replication start for Partition 0?

A Step 2

B Step 6

C Step 3

D Step 7

💡 Hint

Look for 'Start replication' action for Partition 0 in the execution table.

If the replication factor was set to 4 but only 3 brokers exist, what would happen?

A Topic creation succeeds with 4 replicas on same brokers

B Replication factor automatically reduces to 3

C Topic creation fails due to insufficient brokers

D Kafka creates new brokers automatically

💡 Hint

Recall the key moment about replication factor vs number of brokers.

Concept Snapshot

Replication factor sets how many copies of each partition exist.
Each replica is on a different broker for safety.
One replica is leader; others are followers.
Leaders handle client requests and replicate data.
Replication ensures data availability if brokers fail.

Full Transcript

Replication factor in Kafka means how many copies of each partition are stored on different brokers. When creating a topic, Kafka assigns replicas for each partition across brokers and elects a leader for each partition. The leader handles all client reads and writes and replicates data to follower brokers. This protects data if a broker fails. The replication factor cannot be larger than the number of brokers because each replica must be on a separate broker. The execution table shows the step-by-step process from topic creation, replica assignment, leader election, to replication start and client data flow.