Partitioner behavior in Kafka - Step-by-Step Execution

Process Flow - Partitioner behavior

Producer sends message

↓

Check if key is provided

↓

Compute hash of key

↓

Calculate partition number

↓

Send message to chosen partition

The producer decides the partition for each message by checking if a key exists. If yes, it hashes the key to pick a partition; if no, it uses round-robin or default logic.

Execution Sample

Kafka

key = "user123"
num_partitions = 3
partition = hash(key) % num_partitions
send_to_partition(partition)

This code picks a partition for a message with key 'user123' by hashing the key and using modulo with the number of partitions.

Process Table

Step	Action	Input	Computation	Result
1	Producer prepares message	key='user123'	N/A	Message ready with key
2	Check if key exists	key='user123'	key exists	Proceed to hash key
3	Compute hash	key='user123'	hash('user123')	e.g. 123456789
4	Calculate partition	hash=123456789, partitions=3	123456789 % 3	0
5	Send message	partition=0	N/A	Message sent to partition 0
6	Next message without key	key=None	No key	Use round-robin partitioner
7	Round-robin selects partition	partitions=3	next partition in sequence	partition 1
8	Send message	partition=1	N/A	Message sent to partition 1
9	Exit	No more messages	N/A	Producer stops sending

💡 No more messages to send, producer stops execution

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	Final
key	None	"user123"	"user123"	None	None
hash(key)	N/A	123456789	123456789	N/A	N/A
partition	N/A	N/A	0	1	1

Key Moments - 3 Insights

Why does the partition number depend on the key's hash?

Because hashing the key ensures messages with the same key always go to the same partition, as shown in step 3 and 4 of the execution_table.

What happens if the message has no key?

The producer uses a round-robin or default partitioner to distribute messages evenly, as seen in steps 6 and 7.

Can the partition number be greater than the number of partitions?

No, because the modulo operation (step 4) limits the partition number to between 0 and num_partitions-1.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what is the partition number calculated for key 'user123'?

A 1

B 2

C 0

D 3

💡 Hint

Check the 'Result' column at step 4 in the execution_table.

At which step does the producer decide to use round-robin partitioning?

A Step 6

B Step 3

C Step 2

D Step 5

💡 Hint

Look for the step where key is None and round-robin is chosen in the execution_table.

If the number of partitions changes from 3 to 4, how does it affect the partition calculation?

A Partition number stays 0 to 2

B Partition number can now be 0 to 3

C Hash value changes

D Producer stops sending messages

💡 Hint

Refer to the modulo operation in step 4 and how it depends on num_partitions.

Concept Snapshot

Partitioner behavior in Kafka:
- If message has a key, hash the key.
- Use modulo with number of partitions to pick partition.
- If no key, use round-robin or default partitioner.
- Ensures messages with same key go to same partition.
- Partition number always between 0 and num_partitions-1.

Full Transcript

This visual execution shows how Kafka's producer decides which partition to send a message to. First, it checks if the message has a key. If yes, it computes a hash of the key and uses modulo with the number of partitions to find the partition number. This ensures messages with the same key always go to the same partition. If there is no key, the producer uses a round-robin or default partitioner to distribute messages evenly across partitions. The partition number is always between zero and one less than the total partitions. The execution table traces these steps with example values, and the variable tracker shows how key, hash, and partition variables change. Key moments clarify common confusions about hashing, no-key messages, and partition number limits. The quiz tests understanding of partition calculation and behavior changes when partitions count changes.