Partition count strategy in Kafka - Step-by-Step Execution

Process Flow - Partition count strategy

Start: Define topic

↓

Decide number of partitions

↓

Assign partitions to brokers

↓

Producers send messages

↓

Messages routed to partitions

↓

Consumers read from partitions

↓

Scale partitions if needed

↓

End

This flow shows how Kafka topics are created with a set number of partitions, how messages are routed and consumed, and how partition count affects scaling.

Execution Sample

Kafka

kafka-topics --create --topic my-topic --partitions 3 --replication-factor 2

Producer sends messages with key
Messages go to partition by key hash
Consumer reads from assigned partitions

This example creates a topic with 3 partitions and shows how messages are distributed and consumed.

Process Table

Step	Action	Partition Count	Message Key	Partition Chosen	Consumer Assigned
1	Create topic	3	-	-	-
2	Producer sends message with key 'A'	3	A	Partition 1 (hash of 'A')	-
3	Producer sends message with key 'B'	3	B	Partition 0 (hash of 'B')	-
4	Producer sends message with no key	3	None	Partition 2 (round robin)	-
5	Consumer 1 assigned partitions	3	-	-	Partition 0 and 1
6	Consumer 2 assigned partitions	3	-	-	Partition 2
7	Scale partitions to 5	5	-	-	-
8	Producer sends message with key 'C'	5	C	Partition 3 (hash of 'C')	-
9	Consumer 1 assigned partitions	5	-	-	Partition 0,1,3
10	Consumer 2 assigned partitions	5	-	-	Partition 2,4
11	End	-	-	-	-

💡 Execution stops after scaling partitions and reassigning consumers.

Status Tracker

Variable	Start	After Step 1	After Step 7	Final
Partition Count	Undefined	3	5	5
Message Key	None	Varies per message	Varies per message	Varies per message
Consumer Assignment	None	Partitions split among consumers	Partitions split among consumers	Partitions split among consumers

Key Moments - 3 Insights

Why does a message without a key go to partitions in round robin?

Because without a key, Kafka uses round robin to distribute messages evenly, as shown in execution_table row 4.

What happens to consumer partition assignment when partitions increase?

Consumers get reassigned partitions to balance load, as seen in rows 9 and 10 after scaling to 5 partitions.

Why is the partition count important when creating a topic?

Partition count determines parallelism and scalability; more partitions allow more consumers to read in parallel, shown in step 1 and step 7.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the partition count change?

A Step 1

B Step 7

C Step 4

D Step 10

💡 Hint

Check the 'Partition Count' column in execution_table rows.

At step 4, which partition does a message without a key go to?

A Partition 2

B Partition 1

C Partition 0

D Partition 3

💡 Hint

Look at the 'Partition Chosen' column for step 4 in execution_table.

If the partition count was 2 instead of 3 at step 1, how would consumer assignment change?

A No change in assignment

B Consumers would have more partitions to read

C Consumers would have fewer partitions to read

D Consumers would read from all partitions

💡 Hint

Refer to variable_tracker 'Partition Count' and consumer assignment changes.

Concept Snapshot

Partition count defines how many parts a Kafka topic has.
Messages with keys go to partitions by hashing the key.
Messages without keys are distributed round robin.
Consumers read from assigned partitions.
Increasing partitions allows more parallelism but requires reassignment.

Full Transcript

This visual execution shows how Kafka's partition count strategy works. First, a topic is created with a set number of partitions. Producers send messages with or without keys. Messages with keys are routed to partitions based on a hash of the key, ensuring the same key always goes to the same partition. Messages without keys are distributed evenly using round robin. Consumers are assigned partitions to read from, balancing the load. When partitions increase, consumers get reassigned to cover the new partitions. This strategy helps Kafka scale and balance message processing efficiently.