Partition count strategy in Kafka - Commands & Configuration

Introduction

Kafka splits data into parts called partitions to spread the load and keep things fast. Choosing how many partitions to use helps balance speed and resource use.

When you want to increase how many messages Kafka can handle at the same time.

When you need to make sure your data is spread evenly across servers.

When you want to add more consumers to read data in parallel.

When you want to control how data is ordered within a topic.

When you plan to scale your Kafka cluster smoothly as your app grows.

Config File - topic-config.json

topic-config.json

{
  "TopicName": "example-topic",
  "Partitions": 4,
  "ReplicationFactor": 2
}

This JSON file sets the topic name, the number of partitions, and the replication factor.

Partitions controls how many parts the topic is split into.

ReplicationFactor controls how many copies of each partition exist for safety.

Commands

This command creates a new Kafka topic named 'example-topic' with 4 partitions and 2 copies for fault tolerance.

Terminal

kafka-topics --create --topic example-topic --partitions 4 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets how many partitions the topic will have

→

--replication-factor - Sets how many copies of each partition exist

→

--bootstrap-server - Specifies the Kafka server to connect to

This command shows details about the topic, including the number of partitions and their leaders.

Terminal

kafka-topics --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 4	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

	Topic: example-topic	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2

→

--describe - Shows detailed info about the topic

→

--topic - Specifies which topic to describe

→

--bootstrap-server - Specifies the Kafka server to connect to

This command increases the number of partitions from 4 to 6 to allow more parallel processing.

Terminal

kafka-topics --alter --topic example-topic --partitions 6 --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--alter - Modifies an existing topic

→

--partitions - Sets the new number of partitions

→

--bootstrap-server - Specifies the Kafka server to connect to

Verify the topic now has 6 partitions after the change.

Terminal

kafka-topics --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic	PartitionCount: 6	ReplicationFactor: 2	Configs: 

	Topic: example-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 1	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 2	Leader: 3	Replicas: 3,1	Isr: 3,1

	Topic: example-topic	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2

	Topic: example-topic	Partition: 4	Leader: 2	Replicas: 2,3	Isr: 2,3

	Topic: example-topic	Partition: 5	Leader: 3	Replicas: 3,1	Isr: 3,1

→

--describe - Shows detailed info about the topic

→

--topic - Specifies which topic to describe

→

--bootstrap-server - Specifies the Kafka server to connect to

Key Concept

If you remember nothing else from this pattern, remember: the number of partitions controls how much your Kafka topic can handle in parallel and affects data ordering.

Common Mistakes

Trying to decrease the number of partitions after topic creation

Kafka does not allow reducing partitions because it can cause data loss and ordering issues

Plan the partition count carefully before creating the topic; you can only increase partitions later

Setting too many partitions without enough brokers

This overloads brokers and can reduce performance instead of improving it

Match partition count to the number of brokers and expected load

Not verifying the topic details after creation or alteration

You might miss errors or misconfigurations that affect your app

Always run 'kafka-topics --describe' to confirm your changes

Summary

Create a Kafka topic with a set number of partitions to control parallelism.

Use 'kafka-topics --describe' to check the topic's partition count and status.

Increase partitions with 'kafka-topics --alter' when you need more parallel processing.

Remember you cannot reduce partitions once set, so plan ahead.

Practice

1. What does the partition count in a Kafka topic control?

easy

A. The number of parts a topic is divided into for parallel processing

B. The number of messages a topic can hold

C. The size limit of each message in the topic

D. The number of consumers allowed to read the topic

Solution

1. Step 1: Understand Kafka topic structure

   A Kafka topic is divided into partitions to allow parallel processing of messages.

2. Step 2: Identify what partition count controls

   The partition count sets how many parts the topic is split into, enabling multiple consumers to read in parallel.

3. Final Answer:

   The number of parts a topic is divided into for parallel processing -> Option A

4. Quick Check:

   Partition count = number of topic parts [OK]

Hint: Partition count means how many parts a topic has [OK]

Common Mistakes:

• Confusing partition count with message size
• Thinking partition count limits consumer numbers
• Assuming partition count controls message retention

2. Which of the following is the correct way to set the partition count when creating a Kafka topic using the command line?

easy

A. kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092

B. kafka-topics --create --topic myTopic --num-partitions 3 --bootstrap-server localhost:9092

C. kafka-topics --create --topic myTopic --partition-count 3 --bootstrap-server localhost:9092

D. kafka-topics --create --topic myTopic --part 3 --bootstrap-server localhost:9092

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct flag to set partitions is --partitions.

2. Step 2: Match the correct command option

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 uses --partitions 3, which is the correct syntax.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Partition flag = --partitions [OK]

Hint: Use --partitions to set partition count [OK]

Common Mistakes:

• Using --num-partitions instead of --partitions
• Using --partition-count which is invalid
• Using --part which is invalid

3. Given a Kafka topic with 4 partitions, what will be the partition number for a message with key hash % 4 = 2?

medium

A. Partition 0

B. Partition 1

C. Partition 2

D. Partition 3

Solution

1. Step 1: Understand partition assignment by key hash

   Kafka assigns messages to partitions by taking the key hash modulo the number of partitions.

2. Step 2: Calculate partition number

   Given key hash % 4 = 2, the message goes to partition 2 (zero-based indexing).

3. Final Answer:

   Partition 2 -> Option C

4. Quick Check:

   Key hash % partitions = partition number [OK]

Hint: Partition = key hash modulo partition count [OK]

Common Mistakes:

• Confusing modulo result with partition index
• Assuming partitions start at 1 instead of 0
• Ignoring key hash in partition calculation

4. You tried to create a Kafka topic with 0 partitions using the command: kafka-topics --create --topic test --num-partitions 0 --bootstrap-server localhost:9092. What is the likely issue?

medium

A. Kafka allows 0 partitions, so topic created with no data storage

B. Error because partition count must be at least 1

C. Topic created with default partition count of 1

D. Command ignored the partition count and created topic anyway

Solution

1. Step 1: Understand partition count constraints

   Kafka requires at least one partition per topic; zero is invalid.

2. Step 2: Identify command behavior

   Setting partitions to 0 causes an error because Kafka cannot create a topic without partitions.

3. Final Answer:

   Error because partition count must be at least 1 -> Option B

4. Quick Check:

   Partition count ≥ 1 required [OK]

Hint: Partition count cannot be zero or negative [OK]

Common Mistakes:

• Assuming Kafka defaults to 1 partition silently
• Thinking zero partitions means no data but topic exists
• Ignoring error messages from Kafka CLI

5. You have a Kafka topic with 5 partitions and want to increase parallel processing. Which is the best strategy?

hard

A. Delete the topic and create a new one with 1 partition

B. Decrease the partition count to 3 to reduce complexity

C. Keep partition count at 5 and add more consumers than partitions

D. Increase the partition count to 10 and rebalance consumers

Solution

1. Step 1: Understand effect of partition count on parallelism

   More partitions allow more consumers to read in parallel, increasing throughput.

2. Step 2: Evaluate options for increasing parallel processing

   Increasing partitions to 10 allows better parallelism; rebalance consumers to use new partitions.

3. Final Answer:

   Increase the partition count to 10 and rebalance consumers -> Option D

4. Quick Check:

   More partitions = better parallel processing [OK]

Hint: More partitions mean more parallel consumers [OK]

Common Mistakes:

• Reducing partitions thinking it improves speed
• Adding more consumers than partitions without increasing partitions
• Deleting topic unnecessarily