Replication factor in Kafka - Step-by-Step Execution

Process Flow - Replication factor

Create Topic Request

↓

Assign Replicas per Partition

↓

Distribute Replicas Across Brokers

↓

Start Replication

↓

Leader Election for Each Partition

↓

Clients Produce/Consume Data

↓

Replication Maintains Data Copies

When a topic is created, Kafka assigns multiple copies (replicas) of each partition across brokers to ensure data safety and availability.

Execution Sample

Kafka

kafka-topics.sh --create --topic my-topic --partitions 2 --replication-factor 3 --bootstrap-server localhost:9092

This command creates a topic named 'my-topic' with 2 partitions and 3 replicas for each partition.

Process Table

Step	Action	Partition	Replicas Assigned	Leader Broker	Notes
1	Create topic request received	-	-	-	Topic 'my-topic' creation starts
2	Assign replicas	Partition 0	Broker 1, Broker 2, Broker 3	-	Replicas spread across brokers
3	Assign replicas	Partition 1	Broker 2, Broker 3, Broker 1	-	Replicas assigned for partition 1
4	Leader election	Partition 0	Broker 1, Broker 2, Broker 3	Broker 1	Leader chosen for partition 0
5	Leader election	Partition 1	Broker 2, Broker 3, Broker 1	Broker 2	Leader chosen for partition 1
6	Start replication	Partition 0	Broker 1, Broker 2, Broker 3	Broker 1	Leader replicates data to followers
7	Start replication	Partition 1	Broker 2, Broker 3, Broker 1	Broker 2	Leader replicates data to followers
8	Clients produce/consume	-	-	-	Data is safely stored with replication
9	Exit	-	-	-	Topic created with replication factor 3

💡 All partitions assigned 3 replicas; leaders elected; replication started successfully.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	Final
Partition 0 Replicas	None	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3	Broker 1, Broker 2, Broker 3
Partition 1 Replicas	None	None	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1	Broker 2, Broker 3, Broker 1
Partition 0 Leader	None	None	None	Broker 1	Broker 1	Broker 1	Broker 1	Broker 1
Partition 1 Leader	None	None	None	None	Broker 2	Broker 2	Broker 2	Broker 2

Key Moments - 3 Insights

Why does each partition have multiple replicas?

Each partition has multiple replicas to keep copies of data on different brokers. This protects data if one broker fails, as shown in steps 2 and 3 where replicas are assigned.

What is the role of the leader broker for a partition?

The leader broker handles all read and write requests for its partition and replicates data to follower brokers. This is clear in steps 6 and 7 where leaders start replication.

Can replication factor be larger than the number of brokers?

No, replication factor cannot exceed the number of brokers because each replica must be on a different broker. This is why in step 2 replicas are assigned to distinct brokers.

Visual Quiz - 3 Questions

Look at the execution table, which broker is the leader for Partition 1 after assignment?

A No leader assigned yet

B Broker 1

C Broker 3

D Broker 2

💡 Hint

Check the 'Leader Broker' column for Partition 1 in step 3.

At which step does replication start for Partition 0?

A Step 2

B Step 6

C Step 3

D Step 7

💡 Hint

Look for 'Start replication' action for Partition 0 in the execution table.

If the replication factor was set to 4 but only 3 brokers exist, what would happen?

A Topic creation succeeds with 4 replicas on same brokers

B Replication factor automatically reduces to 3

C Topic creation fails due to insufficient brokers

D Kafka creates new brokers automatically

💡 Hint

Recall the key moment about replication factor vs number of brokers.

Concept Snapshot

Replication factor sets how many copies of each partition exist.
Each replica is on a different broker for safety.
One replica is leader; others are followers.
Leaders handle client requests and replicate data.
Replication ensures data availability if brokers fail.

Full Transcript

Replication factor in Kafka means how many copies of each partition are stored on different brokers. When creating a topic, Kafka assigns replicas for each partition across brokers and elects a leader for each partition. The leader handles all client reads and writes and replicates data to follower brokers. This protects data if a broker fails. The replication factor cannot be larger than the number of brokers because each replica must be on a separate broker. The execution table shows the step-by-step process from topic creation, replica assignment, leader election, to replication start and client data flow.

Practice

1.

What does the replication factor in Kafka control?

easy

A. The size limit of each partition

B. The number of copies of each partition stored across brokers

C. The number of partitions in a topic

D. The number of consumers in a consumer group

Solution

1. Step 1: Understand replication factor meaning

   The replication factor defines how many copies of each partition Kafka keeps on different brokers.

2. Step 2: Differentiate from other Kafka settings

   Partitions count and consumer group size are unrelated to replication factor.

3. Final Answer:

   The number of copies of each partition stored across brokers -> Option B

4. Quick Check:

   Replication factor = number of copies [OK]

Hint: Replication factor = copies of data stored [OK]

Common Mistakes:

• Confusing replication factor with partition count
• Thinking replication factor controls consumer groups
• Assuming replication factor limits partition size

2.

Which of the following is the correct way to set the replication factor when creating a Kafka topic using the command line?

kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor ? --bootstrap-server localhost:9092

easy

A. --replication-factor 3

B. --replication-factor=3

C. --replication_factor 3

D. --replicationFactor 3

Solution

1. Step 1: Check Kafka CLI syntax for replication factor

   The correct flag is --replication-factor with a space before the value.

2. Step 2: Identify incorrect options

   Options with underscores or camelCase are invalid flags; equals sign is not used here.

3. Final Answer:

   --replication-factor 3 -> Option A

4. Quick Check:

   Use double dash and hyphen for replication factor [OK]

Hint: Use double dash and hyphen for flags in Kafka CLI [OK]

Common Mistakes:

• Using underscore instead of hyphen
• Using equals sign instead of space
• Using camelCase flag names

3.

Given a Kafka topic created with --partitions 2 and --replication-factor 3, how many total replicas will Kafka store across the cluster?

medium

A. 6

B. 3

C. 5

D. 2

Solution

1. Step 1: Calculate replicas per partition

   Each partition has 3 replicas because replication factor is 3.

2. Step 2: Multiply by number of partitions

   2 partitions x 3 replicas each = 6 total replicas.

3. Final Answer:

   6 -> Option A

4. Quick Check:

   Partitions x Replication factor = Total replicas [OK]

Hint: Multiply partitions by replication factor [OK]

Common Mistakes:

• Confusing replication factor with total partitions
• Adding instead of multiplying partitions and replicas
• Ignoring replication factor in calculation

4.

What is wrong with this Kafka topic creation command?

kafka-topics.sh --create --topic logs --partitions 3 --replication-factor 5 --bootstrap-server localhost:9092

Assuming the Kafka cluster has only 3 brokers.

medium

A. Partitions must be equal to replication factor

B. Topic name cannot be 'logs'

C. Bootstrap server address is incorrect

D. Replication factor cannot be greater than the number of brokers

Solution

1. Step 1: Understand replication factor limits

   Replication factor must be less than or equal to the number of brokers to ensure each replica is on a different broker.

2. Step 2: Check cluster size vs replication factor

   Replication factor 5 is greater than 3 brokers, which is invalid.

3. Final Answer:

   Replication factor cannot be greater than the number of brokers -> Option D

4. Quick Check:

   Replication factor ≤ brokers [OK]

Hint: Replication factor must not exceed broker count [OK]

Common Mistakes:

• Thinking partitions must equal replication factor
• Ignoring cluster size limits
• Assuming topic names have restrictions here

5.

You want to create a Kafka topic with 4 partitions and a replication factor of 3 for high availability. However, your cluster has only 2 brokers. What is the best solution?

hard

A. Create the topic with 1 partition and replication factor 3

B. Set replication factor to 4 to match partitions

C. Increase the number of brokers to at least 3

D. Create the topic with replication factor 2 instead

Solution

1. Step 1: Check replication factor vs brokers

   Replication factor 3 requires at least 3 brokers to store replicas on different machines.

2. Step 2: Evaluate options

   Reducing replication factor lowers availability; increasing brokers meets requirement.

3. Final Answer:

   Increase the number of brokers to at least 3 -> Option C

4. Quick Check:

   Replication factor ≤ brokers for high availability [OK]

Hint: Match replication factor to broker count for safety [OK]

Common Mistakes:

• Setting replication factor higher than brokers
• Reducing partitions instead of brokers
• Ignoring availability requirements