Partition count strategy in Kafka - Commands & Configuration

Introduction

Kafka splits data into parts called partitions to spread the load and keep things fast. Choosing how many partitions to use helps balance speed and resource use.

When you want to increase how many messages Kafka can handle at the same time.

When you need to make sure your data is spread evenly across servers.

When you want to add more consumers to read data in parallel.

When you want to control how data is ordered within a topic.

When you plan to scale your Kafka cluster smoothly as your app grows.

Config File - topic-config.json

topic-config.json

{
  "TopicName": "example-topic",
  "Partitions": 4,
  "ReplicationFactor": 2
}

This JSON file sets the topic name, the number of partitions, and the replication factor.

Partitions controls how many parts the topic is split into.

ReplicationFactor controls how many copies of each partition exist for safety.

Commands

This command creates a new Kafka topic named 'example-topic' with 4 partitions and 2 copies for fault tolerance.

Terminal

kafka-topics --create --topic example-topic --partitions 4 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets how many partitions the topic will have

→

--replication-factor - Sets how many copies of each partition exist

→

--bootstrap-server - Specifies the Kafka server to connect to

This command shows details about the topic, including the number of partitions and their leaders.

Terminal

kafka-topics --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 4	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

	Topic: example-topic	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2

→

--describe - Shows detailed info about the topic

→

--topic - Specifies which topic to describe

→

--bootstrap-server - Specifies the Kafka server to connect to

This command increases the number of partitions from 4 to 6 to allow more parallel processing.

Terminal

kafka-topics --alter --topic example-topic --partitions 6 --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--alter - Modifies an existing topic

→

--partitions - Sets the new number of partitions

→

--bootstrap-server - Specifies the Kafka server to connect to

Verify the topic now has 6 partitions after the change.

Terminal

kafka-topics --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 6	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

	Topic: example-topic	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 4	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 5	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--describe - Shows detailed info about the topic

→

--topic - Specifies which topic to describe

→

--bootstrap-server - Specifies the Kafka server to connect to

Key Concept

If you remember nothing else from this pattern, remember: the number of partitions controls how much your Kafka topic can handle in parallel and affects data ordering.

Common Mistakes

Trying to decrease the number of partitions after topic creation

Kafka does not allow reducing partitions because it can cause data loss and ordering issues

Plan the partition count carefully before creating the topic; you can only increase partitions later

Setting too many partitions without enough brokers

This overloads brokers and can reduce performance instead of improving it

Match partition count to the number of brokers and expected load

Not verifying the topic details after creation or alteration

You might miss errors or misconfigurations that affect your app

Always run 'kafka-topics --describe' to confirm your changes

Summary

Create a Kafka topic with a set number of partitions to control parallelism.

Use 'kafka-topics --describe' to check the topic's partition count and status.

Increase partitions with 'kafka-topics --alter' when you need more parallel processing.

Remember you cannot reduce partitions once set, so plan ahead.