Why Partition count strategy in Kafka? - Purpose & Use Cases

The Big Idea

What if your message system could handle millions of requests without slowing down?

The Scenario

Imagine you have a busy post office with only one counter to handle all customers. Everyone waits in a long line, and the staff struggles to keep up with the flood of letters and packages.

The Problem

Handling all messages through a single partition is like that one counter: slow, easily overwhelmed, and prone to delays. If the load grows, the system becomes a bottleneck, causing slow processing and unhappy users.

The Solution

Using a partition count strategy means splitting the workload across multiple counters (partitions). Each counter handles a portion of the messages, allowing many to be processed at the same time, making the system faster and more reliable.

Before vs After

✗ Before

topic = kafka.create_topic('orders', num_partitions=1)

✓ After

topic = kafka.create_topic('orders', num_partitions=5)

What It Enables

This strategy enables smooth scaling and faster message processing by distributing work evenly across partitions.

Real Life Example

Think of a busy supermarket with many checkout lanes open instead of just one. Customers move through faster, and the store handles more shoppers without chaos.

Key Takeaways

Single partition limits speed and scalability.

Multiple partitions allow parallel processing.

Partition count strategy improves performance and reliability.

Practice

1. What does the partition count in a Kafka topic control?

easy

A. The number of parts a topic is divided into for parallel processing

B. The number of messages a topic can hold

C. The size limit of each message in the topic

D. The number of consumers allowed to read the topic

Solution

1. Step 1: Understand Kafka topic structure

   A Kafka topic is divided into partitions to allow parallel processing of messages.

2. Step 2: Identify what partition count controls

   The partition count sets how many parts the topic is split into, enabling multiple consumers to read in parallel.

3. Final Answer:

   The number of parts a topic is divided into for parallel processing -> Option A

4. Quick Check:

   Partition count = number of topic parts [OK]

Hint: Partition count means how many parts a topic has [OK]

Common Mistakes:

• Confusing partition count with message size
• Thinking partition count limits consumer numbers
• Assuming partition count controls message retention

2. Which of the following is the correct way to set the partition count when creating a Kafka topic using the command line?

easy

A. kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092

B. kafka-topics --create --topic myTopic --num-partitions 3 --bootstrap-server localhost:9092

C. kafka-topics --create --topic myTopic --partition-count 3 --bootstrap-server localhost:9092

D. kafka-topics --create --topic myTopic --part 3 --bootstrap-server localhost:9092

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct flag to set partitions is --partitions.

2. Step 2: Match the correct command option

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 uses --partitions 3, which is the correct syntax.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Partition flag = --partitions [OK]

Hint: Use --partitions to set partition count [OK]

Common Mistakes:

• Using --num-partitions instead of --partitions
• Using --partition-count which is invalid
• Using --part which is invalid

3. Given a Kafka topic with 4 partitions, what will be the partition number for a message with key hash % 4 = 2?

medium

A. Partition 0

B. Partition 1

C. Partition 2

D. Partition 3

Solution

1. Step 1: Understand partition assignment by key hash

   Kafka assigns messages to partitions by taking the key hash modulo the number of partitions.

2. Step 2: Calculate partition number

   Given key hash % 4 = 2, the message goes to partition 2 (zero-based indexing).

3. Final Answer:

   Partition 2 -> Option C

4. Quick Check:

   Key hash % partitions = partition number [OK]

Hint: Partition = key hash modulo partition count [OK]

Common Mistakes:

• Confusing modulo result with partition index
• Assuming partitions start at 1 instead of 0
• Ignoring key hash in partition calculation

4. You tried to create a Kafka topic with 0 partitions using the command: kafka-topics --create --topic test --num-partitions 0 --bootstrap-server localhost:9092. What is the likely issue?

medium

A. Kafka allows 0 partitions, so topic created with no data storage

B. Error because partition count must be at least 1

C. Topic created with default partition count of 1

D. Command ignored the partition count and created topic anyway

Solution

1. Step 1: Understand partition count constraints

   Kafka requires at least one partition per topic; zero is invalid.

2. Step 2: Identify command behavior

   Setting partitions to 0 causes an error because Kafka cannot create a topic without partitions.

3. Final Answer:

   Error because partition count must be at least 1 -> Option B

4. Quick Check:

   Partition count ≥ 1 required [OK]

Hint: Partition count cannot be zero or negative [OK]

Common Mistakes:

• Assuming Kafka defaults to 1 partition silently
• Thinking zero partitions means no data but topic exists
• Ignoring error messages from Kafka CLI

5. You have a Kafka topic with 5 partitions and want to increase parallel processing. Which is the best strategy?

hard

A. Delete the topic and create a new one with 1 partition

B. Decrease the partition count to 3 to reduce complexity

C. Keep partition count at 5 and add more consumers than partitions

D. Increase the partition count to 10 and rebalance consumers

Solution

1. Step 1: Understand effect of partition count on parallelism

   More partitions allow more consumers to read in parallel, increasing throughput.

2. Step 2: Evaluate options for increasing parallel processing

   Increasing partitions to 10 allows better parallelism; rebalance consumers to use new partitions.

3. Final Answer:

   Increase the partition count to 10 and rebalance consumers -> Option D

4. Quick Check:

   More partitions = better parallel processing [OK]

Hint: More partitions mean more parallel consumers [OK]

Common Mistakes:

• Reducing partitions thinking it improves speed
• Adding more consumers than partitions without increasing partitions
• Deleting topic unnecessarily