Partition key and routing in Kafka - Step-by-Step Execution

Process Flow - Partition key and routing

Message Created

↓

Check for Partition Key?

No→Use Round-Robin Partitioning

Yes↓

Hash Partition Key

↓

Calculate Partition Number

↓

Send Message to Partition

Kafka decides which partition a message goes to by checking if it has a key. If yes, it hashes the key to pick a partition. If no key, it uses round-robin.

Execution Sample

Kafka

producer.send(topic, key=b'user1', value=b'msg1')
producer.send(topic, key=b'user2', value=b'msg2')
producer.send(topic, value=b'msg3')

Send three messages: two with keys (user1, user2) and one without key to a Kafka topic.

Process Table

Step	Message	Has Key?	Key Value	Partition Calculation	Partition Chosen	Action
1	msg1	Yes	user1	hash('user1') % 3 = 1	Partition 1	Send to Partition 1
2	msg2	Yes	user2	hash('user2') % 3 = 0	Partition 0	Send to Partition 0
3	msg3	No	None	Round-Robin next partition = 2	Partition 2	Send to Partition 2
4	End	-	-	-	-	All messages routed

💡 All messages processed and assigned to partitions based on key or round-robin

Status Tracker

Variable	Start	After 1	After 2	After 3	Final
partition	None	1	0	2	2
key	None	user1	user2	None	None
round_robin_counter	0	0	0	2	2

Key Moments - 2 Insights

Why does the message without a key go to Partition 2?

Because it has no key, Kafka uses round-robin to assign partitions. The round-robin counter was at 2 for this message (see execution_table row 3).

How is the partition number calculated for messages with keys?

Kafka hashes the key string and then takes the remainder when divided by the number of partitions (3 here). This is shown in execution_table rows 1 and 2.

Visual Quiz - 3 Questions

Look at the execution table, what partition does the message with key 'user2' go to?

A Partition 1

B Partition 2

C Partition 0

D Partition 3

💡 Hint

Check execution_table row 2 under 'Partition Chosen'

At which step does Kafka use round-robin to assign a partition?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look for 'Has Key?' = No in execution_table

If the topic had 4 partitions instead of 3, how would the partition for 'user1' change?

A It would still be Partition 1

B It would be hash('user1') % 4, possibly different

C It would use round-robin

D It would always be Partition 0

💡 Hint

Partition depends on hash(key) modulo number of partitions (see execution_table rows 1 and 2)

Concept Snapshot

Kafka uses a partition key to decide where to send messages.
If a key exists, Kafka hashes it and picks a partition by modulo operation.
If no key, Kafka uses round-robin to spread messages evenly.
This ensures messages with the same key go to the same partition.
Partitioning helps Kafka scale and keep message order per key.

Full Transcript

This visual shows how Kafka routes messages to partitions. When a message is created, Kafka checks if it has a key. If yes, it hashes the key and calculates the partition number by taking the hash modulo the number of partitions. If no key is present, Kafka uses round-robin to assign the next partition in order. The execution table traces three messages: two with keys 'user1' and 'user2' and one without a key. The first message goes to partition 1, the second to partition 0, and the third uses round-robin to go to partition 2. Variables like partition number, key, and round-robin counter update step by step. Key moments clarify why messages without keys use round-robin and how partition numbers are calculated. The quiz tests understanding of partition assignment and effects of changing partition count. The snapshot summarizes the key rules of Kafka partitioning for quick review.