Partition count strategy in Kafka - Step-by-Step Execution

Process Flow - Partition count strategy

Start: Define topic

↓

Decide number of partitions

↓

Assign partitions to brokers

↓

Producers send messages

↓

Messages routed to partitions

↓

Consumers read from partitions

↓

Scale partitions if needed

↓

End

This flow shows how Kafka topics are created with a set number of partitions, how messages are routed and consumed, and how partition count affects scaling.

Execution Sample

Kafka

kafka-topics --create --topic my-topic --partitions 3 --replication-factor 2

Producer sends messages with key
Messages go to partition by key hash
Consumer reads from assigned partitions

This example creates a topic with 3 partitions and shows how messages are distributed and consumed.

Process Table

Step	Action	Partition Count	Message Key	Partition Chosen	Consumer Assigned
1	Create topic	3	-	-	-
2	Producer sends message with key 'A'	3	A	Partition 1 (hash of 'A')	-
3	Producer sends message with key 'B'	3	B	Partition 0 (hash of 'B')	-
4	Producer sends message with no key	3	None	Partition 2 (round robin)	-
5	Consumer 1 assigned partitions	3	-	-	Partition 0 and 1
6	Consumer 2 assigned partitions	3	-	-	Partition 2
7	Scale partitions to 5	5	-	-	-
8	Producer sends message with key 'C'	5	C	Partition 3 (hash of 'C')	-
9	Consumer 1 assigned partitions	5	-	-	Partition 0,1,3
10	Consumer 2 assigned partitions	5	-	-	Partition 2,4
11	End	-	-	-	-

💡 Execution stops after scaling partitions and reassigning consumers.

Status Tracker

Variable	Start	After Step 1	After Step 7	Final
Partition Count	Undefined	3	5	5
Message Key	None	Varies per message	Varies per message	Varies per message
Consumer Assignment	None	Partitions split among consumers	Partitions split among consumers	Partitions split among consumers

Key Moments - 3 Insights

Why does a message without a key go to partitions in round robin?

Because without a key, Kafka uses round robin to distribute messages evenly, as shown in execution_table row 4.

What happens to consumer partition assignment when partitions increase?

Consumers get reassigned partitions to balance load, as seen in rows 9 and 10 after scaling to 5 partitions.

Why is the partition count important when creating a topic?

Partition count determines parallelism and scalability; more partitions allow more consumers to read in parallel, shown in step 1 and step 7.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the partition count change?

A Step 1

B Step 7

C Step 4

D Step 10

💡 Hint

Check the 'Partition Count' column in execution_table rows.

At step 4, which partition does a message without a key go to?

A Partition 2

B Partition 1

C Partition 0

D Partition 3

💡 Hint

Look at the 'Partition Chosen' column for step 4 in execution_table.

If the partition count was 2 instead of 3 at step 1, how would consumer assignment change?

A No change in assignment

B Consumers would have more partitions to read

C Consumers would have fewer partitions to read

D Consumers would read from all partitions

💡 Hint

Refer to variable_tracker 'Partition Count' and consumer assignment changes.

Concept Snapshot

Partition count defines how many parts a Kafka topic has.
Messages with keys go to partitions by hashing the key.
Messages without keys are distributed round robin.
Consumers read from assigned partitions.
Increasing partitions allows more parallelism but requires reassignment.

Full Transcript

This visual execution shows how Kafka's partition count strategy works. First, a topic is created with a set number of partitions. Producers send messages with or without keys. Messages with keys are routed to partitions based on a hash of the key, ensuring the same key always goes to the same partition. Messages without keys are distributed evenly using round robin. Consumers are assigned partitions to read from, balancing the load. When partitions increase, consumers get reassigned to cover the new partitions. This strategy helps Kafka scale and balance message processing efficiently.

Practice

1. What does the partition count in a Kafka topic control?

easy

A. The number of parts a topic is divided into for parallel processing

B. The number of messages a topic can hold

C. The size limit of each message in the topic

D. The number of consumers allowed to read the topic

Solution

1. Step 1: Understand Kafka topic structure

   A Kafka topic is divided into partitions to allow parallel processing of messages.

2. Step 2: Identify what partition count controls

   The partition count sets how many parts the topic is split into, enabling multiple consumers to read in parallel.

3. Final Answer:

   The number of parts a topic is divided into for parallel processing -> Option A

4. Quick Check:

   Partition count = number of topic parts [OK]

Hint: Partition count means how many parts a topic has [OK]

Common Mistakes:

• Confusing partition count with message size
• Thinking partition count limits consumer numbers
• Assuming partition count controls message retention

2. Which of the following is the correct way to set the partition count when creating a Kafka topic using the command line?

easy

A. kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092

B. kafka-topics --create --topic myTopic --num-partitions 3 --bootstrap-server localhost:9092

C. kafka-topics --create --topic myTopic --partition-count 3 --bootstrap-server localhost:9092

D. kafka-topics --create --topic myTopic --part 3 --bootstrap-server localhost:9092

Solution

1. Step 1: Recall Kafka topic creation syntax

   The correct flag to set partitions is --partitions.

2. Step 2: Match the correct command option

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 uses --partitions 3, which is the correct syntax.

3. Final Answer:

   kafka-topics --create --topic myTopic --partitions 3 --bootstrap-server localhost:9092 -> Option A

4. Quick Check:

   Partition flag = --partitions [OK]

Hint: Use --partitions to set partition count [OK]

Common Mistakes:

• Using --num-partitions instead of --partitions
• Using --partition-count which is invalid
• Using --part which is invalid

3. Given a Kafka topic with 4 partitions, what will be the partition number for a message with key hash % 4 = 2?

medium

A. Partition 0

B. Partition 1

C. Partition 2

D. Partition 3

Solution

1. Step 1: Understand partition assignment by key hash

   Kafka assigns messages to partitions by taking the key hash modulo the number of partitions.

2. Step 2: Calculate partition number

   Given key hash % 4 = 2, the message goes to partition 2 (zero-based indexing).

3. Final Answer:

   Partition 2 -> Option C

4. Quick Check:

   Key hash % partitions = partition number [OK]

Hint: Partition = key hash modulo partition count [OK]

Common Mistakes:

• Confusing modulo result with partition index
• Assuming partitions start at 1 instead of 0
• Ignoring key hash in partition calculation

4. You tried to create a Kafka topic with 0 partitions using the command: kafka-topics --create --topic test --num-partitions 0 --bootstrap-server localhost:9092. What is the likely issue?

medium

A. Kafka allows 0 partitions, so topic created with no data storage

B. Error because partition count must be at least 1

C. Topic created with default partition count of 1

D. Command ignored the partition count and created topic anyway

Solution

1. Step 1: Understand partition count constraints

   Kafka requires at least one partition per topic; zero is invalid.

2. Step 2: Identify command behavior

   Setting partitions to 0 causes an error because Kafka cannot create a topic without partitions.

3. Final Answer:

   Error because partition count must be at least 1 -> Option B

4. Quick Check:

   Partition count ≥ 1 required [OK]

Hint: Partition count cannot be zero or negative [OK]

Common Mistakes:

• Assuming Kafka defaults to 1 partition silently
• Thinking zero partitions means no data but topic exists
• Ignoring error messages from Kafka CLI

5. You have a Kafka topic with 5 partitions and want to increase parallel processing. Which is the best strategy?

hard

A. Delete the topic and create a new one with 1 partition

B. Decrease the partition count to 3 to reduce complexity

C. Keep partition count at 5 and add more consumers than partitions

D. Increase the partition count to 10 and rebalance consumers

Solution

1. Step 1: Understand effect of partition count on parallelism

   More partitions allow more consumers to read in parallel, increasing throughput.

2. Step 2: Evaluate options for increasing parallel processing

   Increasing partitions to 10 allows better parallelism; rebalance consumers to use new partitions.

3. Final Answer:

   Increase the partition count to 10 and rebalance consumers -> Option D

4. Quick Check:

   More partitions = better parallel processing [OK]

Hint: More partitions mean more parallel consumers [OK]

Common Mistakes:

• Reducing partitions thinking it improves speed
• Adding more consumers than partitions without increasing partitions
• Deleting topic unnecessarily