Partition concept in Kafka - Commands & Configuration

Introduction

Kafka partitions split a topic into smaller parts so messages can be stored and processed in parallel. This helps handle more data and keeps the system fast and reliable.

When you want to increase the speed of message processing by spreading load across multiple servers.

When you need to keep the order of messages for a specific key but allow parallel processing for different keys.

When your topic receives more messages than a single server can handle efficiently.

When you want to scale your Kafka cluster by adding more brokers and distributing partitions among them.

When you want to improve fault tolerance by replicating partitions across different servers.

Config File - topic-config.json

topic-config.json

{
  "TopicName": "example-topic",
  "Partitions": 3,
  "ReplicationFactor": 2
}

This JSON file defines a Kafka topic named example-topic with 3 partitions. Each partition allows parallel processing of messages. The ReplicationFactor of 2 means each partition is copied to two brokers for safety.

Commands

This command creates a Kafka topic named example-topic with 3 partitions and a replication factor of 2. Partitions allow parallel processing and replication ensures data safety.

Terminal

kafka-topics.sh --create --topic example-topic --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets the number of partitions for the topic

→

--replication-factor - Sets how many copies of each partition exist

→

--bootstrap-server - Specifies the Kafka server to connect to

This command shows details about the topic, including the number of partitions and their leaders. It helps verify the topic was created correctly.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 3	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--describe - Shows detailed information about the topic

→

--topic - Specifies which topic to describe

→

--bootstrap-server - Specifies the Kafka server to connect to

Key Concept

Partitions split a topic so messages can be processed in parallel and scaled across servers.

Common Mistakes

Creating a topic with only one partition when expecting high message volume

One partition limits processing to a single server, causing slow performance and no parallelism.

Create the topic with multiple partitions to allow parallel processing and better scalability.

Setting replication factor higher than the number of brokers

Kafka cannot create more replicas than available brokers, causing topic creation to fail.

Ensure the replication factor is less than or equal to the number of Kafka brokers in the cluster.

Summary

Create Kafka topics with multiple partitions to enable parallel message processing.

Use replication factor to keep copies of partitions for fault tolerance.

Verify topic details with the describe command to check partitions and leaders.