Replication factor in Kafka - Commands & Configuration

Introduction

When you run Kafka, you want your messages to be safe even if some servers fail. Replication factor tells Kafka how many copies of your data it should keep on different servers. This helps keep your data safe and available.

When you want to make sure your messages are not lost if one Kafka server crashes

When you run Kafka on multiple servers and want to spread copies of data for safety

When you want to balance between data safety and storage space used

When you want to improve Kafka availability during maintenance or unexpected failures

When you want to control how many servers share the same data for fault tolerance

Commands

This command creates a new Kafka topic named 'example-topic' with 3 partitions and a replication factor of 2. This means each partition will have 2 copies on different Kafka servers for safety.

Terminal

kafka-topics.sh --create --topic example-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2

Expected Output

Created topic example-topic.

→

--replication-factor - Sets how many copies of each partition Kafka keeps on different servers

→

--partitions - Defines how many parts the topic is split into for parallel processing

This command shows details about the 'example-topic', including its partitions and replication factor. It helps verify that the topic was created with the right replication settings.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

Key Concept

If you remember nothing else from this pattern, remember: replication factor controls how many copies of your Kafka data exist to keep it safe from server failures.

Common Mistakes

Setting replication factor higher than the number of Kafka brokers

Kafka cannot create more copies than the number of servers available, so the topic creation will fail.

Set the replication factor to a number less than or equal to the number of Kafka brokers in your cluster.

Using replication factor of 1 in production

With only one copy, if the server holding the data fails, the data is lost and unavailable.

Use a replication factor of at least 2 or 3 to ensure data safety and availability.

Summary

Create a Kafka topic with the --replication-factor flag to set how many copies of data Kafka keeps.

Use kafka-topics.sh --describe to check the topic's replication factor and partitions.

Replication factor helps keep your data safe by storing copies on multiple servers.