Partition concept in Kafka - Step-by-Step Execution

Process Flow - Partition concept

Message Produced

↓

Partitioner Decides Partition

↓

Message Stored in Partition

↓

Consumer Reads from Partition

↓

Message Processing

A message is produced, the partitioner decides which partition it goes to, the message is stored there, and consumers read messages from partitions for processing.

Execution Sample

Kafka

producer.send(topic, key, value)
// Partitioner uses key to pick partition
// Message stored in that partition
// Consumer reads from assigned partitions

This code sends a message with a key to a Kafka topic; the partitioner uses the key to decide the partition where the message is stored; consumers read messages from partitions.

Process Table

Step	Action	Key	Partition Chosen	Message Stored	Consumer Reads From
1	Produce message	user123	Partition 2	Partition 2	Partition 2
2	Produce message	user456	Partition 0	Partition 0	Partition 0
3	Produce message	user789	Partition 1	Partition 1	Partition 1
4	Produce message	user123	Partition 2	Partition 2	Partition 2
5	Produce message	null	Partition 0 (round-robin)	Partition 0	Partition 0
6	Stop producing	-	-	-	-

💡 No more messages to produce; processing stops.

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
Key	N/A	user123	user456	user789	user123	null	null
Partition Chosen	N/A	2	0	1	2	0	0
Messages in Partition 0	0	0	1	1	1	2	2
Messages in Partition 1	0	0	0	1	1	1	1
Messages in Partition 2	0	1	1	1	2	2	2

Key Moments - 2 Insights

Why does the same key always go to the same partition?

Because the partitioner uses the key's hash to pick a partition, so the same key maps to the same partition every time, as shown in steps 1 and 4 in the execution_table.

What happens when the key is null?

When the key is null, Kafka uses round-robin to assign partitions, so messages without keys are distributed evenly, as seen in step 5.

Visual Quiz - 3 Questions

Look at the execution_table, which partition does the message with key 'user789' go to?

A Partition 0

B Partition 1

C Partition 2

D Partition 3

💡 Hint

Check step 3 in the execution_table under 'Partition Chosen'.

At which step does the partitioner assign a partition using round-robin because the key is null?

A Step 2

B Step 4

C Step 5

D Step 6

💡 Hint

Look for the row where the key is 'null' in the execution_table.

If the key 'user123' changed to 'user999', what would likely happen to the partition chosen?

A It would change to a different partition

B It would stay the same partition 2

C It would cause an error

D It would be assigned randomly every time

💡 Hint

Since partition depends on key hash, changing key changes partition, see variable_tracker for key and partition relation.

Concept Snapshot

Kafka Partition Concept:
- Messages are sent to topics divided into partitions.
- Partitioner uses message key to pick partition (same key → same partition).
- Messages without keys are assigned partitions round-robin.
- Consumers read messages from assigned partitions.
- Partitions enable parallelism and ordering per key.

Full Transcript

In Kafka, when a message is produced, the partitioner decides which partition the message goes to based on the message key. The same key always maps to the same partition, ensuring order for that key. If the key is null, Kafka assigns partitions in a round-robin way to balance load. Consumers read messages from partitions to process them. This flow helps Kafka handle many messages efficiently and keep order per key.