Partitioner behavior in Kafka - Step-by-Step Execution

Process Flow - Partitioner behavior

Producer sends message

↓

Check if key is provided

↓

Compute hash of key

↓

Calculate partition number

↓

Send message to chosen partition

The producer decides the partition for each message by checking if a key exists. If yes, it hashes the key to pick a partition; if no, it uses round-robin or default logic.

Execution Sample

Kafka

key = "user123"
num_partitions = 3
partition = hash(key) % num_partitions
send_to_partition(partition)

This code picks a partition for a message with key 'user123' by hashing the key and using modulo with the number of partitions.

Process Table

Step	Action	Input	Computation	Result
1	Producer prepares message	key='user123'	N/A	Message ready with key
2	Check if key exists	key='user123'	key exists	Proceed to hash key
3	Compute hash	key='user123'	hash('user123')	e.g. 123456789
4	Calculate partition	hash=123456789, partitions=3	123456789 % 3	0
5	Send message	partition=0	N/A	Message sent to partition 0
6	Next message without key	key=None	No key	Use round-robin partitioner
7	Round-robin selects partition	partitions=3	next partition in sequence	partition 1
8	Send message	partition=1	N/A	Message sent to partition 1
9	Exit	No more messages	N/A	Producer stops sending

💡 No more messages to send, producer stops execution

Status Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	Final
key	None	"user123"	"user123"	None	None
hash(key)	N/A	123456789	123456789	N/A	N/A
partition	N/A	N/A	0	1	1

Key Moments - 3 Insights

Why does the partition number depend on the key's hash?

Because hashing the key ensures messages with the same key always go to the same partition, as shown in step 3 and 4 of the execution_table.

What happens if the message has no key?

The producer uses a round-robin or default partitioner to distribute messages evenly, as seen in steps 6 and 7.

Can the partition number be greater than the number of partitions?

No, because the modulo operation (step 4) limits the partition number to between 0 and num_partitions-1.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what is the partition number calculated for key 'user123'?

A 1

B 2

C 0

D 3

💡 Hint

Check the 'Result' column at step 4 in the execution_table.

At which step does the producer decide to use round-robin partitioning?

A Step 6

B Step 3

C Step 2

D Step 5

💡 Hint

Look for the step where key is None and round-robin is chosen in the execution_table.

If the number of partitions changes from 3 to 4, how does it affect the partition calculation?

A Partition number stays 0 to 2

B Partition number can now be 0 to 3

C Hash value changes

D Producer stops sending messages

💡 Hint

Refer to the modulo operation in step 4 and how it depends on num_partitions.

Concept Snapshot

Partitioner behavior in Kafka:
- If message has a key, hash the key.
- Use modulo with number of partitions to pick partition.
- If no key, use round-robin or default partitioner.
- Ensures messages with same key go to same partition.
- Partition number always between 0 and num_partitions-1.

Full Transcript

This visual execution shows how Kafka's producer decides which partition to send a message to. First, it checks if the message has a key. If yes, it computes a hash of the key and uses modulo with the number of partitions to find the partition number. This ensures messages with the same key always go to the same partition. If there is no key, the producer uses a round-robin or default partitioner to distribute messages evenly across partitions. The partition number is always between zero and one less than the total partitions. The execution table traces these steps with example values, and the variable tracker shows how key, hash, and partition variables change. Key moments clarify common confusions about hashing, no-key messages, and partition number limits. The quiz tests understanding of partition calculation and behavior changes when partitions count changes.

Practice

1.

What does a Kafka partitioner do?

partitioner decides:

easy

A. Which partition a message is sent to

B. How to compress messages

C. The message format

D. The topic name

Solution

1. Step 1: Understand the role of a partitioner

   A partitioner decides the partition inside a Kafka topic where a message will be stored.

2. Step 2: Match the function to the options

   Only Which partition a message is sent to correctly describes this role.

3. Final Answer:

   Which partition a message is sent to -> Option A

4. Quick Check:

   Partitioner = decides message partition [OK]

Hint: Partitioner chooses partition for messages [OK]

Common Mistakes:

• Confusing partitioner with compression
• Thinking partitioner sets topic name
• Assuming partitioner changes message format

2.

Which of the following is the correct way to specify a default partitioner class in Kafka producer configuration?

props.put("partitioner.class", ?);

easy

A. "org.apache.kafka.clients.producer.KeyPartitioner"

B. "org.apache.kafka.clients.producer.RoundRobinPartitioner"

C. "org.apache.kafka.clients.producer.CustomPartitioner"

D. "org.apache.kafka.clients.producer.internals.DefaultPartitioner"

Solution

1. Step 1: Recall default partitioner class name

   The default partitioner class in Kafka is DefaultPartitioner located in org.apache.kafka.clients.producer.internals.

2. Step 2: Match the correct class name

   "org.apache.kafka.clients.producer.internals.DefaultPartitioner" matches the default partitioner class name exactly.

3. Final Answer:

   "org.apache.kafka.clients.producer.internals.DefaultPartitioner" -> Option D

4. Quick Check:

   Default partitioner class = DefaultPartitioner [OK]

Hint: Default partitioner class is DefaultPartitioner [OK]

Common Mistakes:

• Using non-existent class names
• Confusing round-robin with default partitioner
• Misspelling package or class names

3.

Given a Kafka topic with 3 partitions and a producer sending messages with keys "key1", "key2", and "key1" again, which partitions will the messages go to if the default partitioner is used?

medium

A. [0, 0, 1]

B. [1, 2, 0]

C. [0, 1, 0]

D. [2, 1, 0]

Solution

1. Step 1: Understand key-based partitioning

   Messages with the same key go to the same partition. Different keys may go to different partitions.

2. Step 2: Assign partitions for keys

   Assuming hash(key) mod 3, "key1" maps to partition 0, "key2" maps to partition 1. So messages go to partitions [0, 1, 0].

3. Final Answer:

   [0, 1, 0] -> Option C

4. Quick Check:

   Same key = same partition [OK]

Hint: Same key always same partition [OK]

Common Mistakes:

• Assuming keys distribute randomly
• Assigning different partitions to same key
• Ignoring modulo partition count

4.

Consider this code snippet for sending messages without keys:

for i in range(4):
    producer.send("topic", value=f"msg{i}")

What is the likely issue if all messages end up in the same partition?

medium

A. Messages have keys that force same partition

B. Topic has only one partition

C. The producer is not configured to use round-robin partitioning

D. Producer is using a custom partitioner ignoring keys

Solution

1. Step 1: Analyze message keys and partition count

   Messages have no keys, so default partitioner uses round-robin if multiple partitions exist.

2. Step 2: Check topic partition count

   If all messages go to the same partition, likely the topic has only one partition.

3. Final Answer:

   Topic has only one partition -> Option B

4. Quick Check:

   Single partition topic = all messages same partition [OK]

Hint: Single partition topic sends all messages there [OK]

Common Mistakes:

• Assuming round-robin works with one partition
• Thinking keys are present when they are not
• Ignoring topic partition count

5.

You want to ensure messages with the same user ID always go to the same partition, but messages without user ID keys should be evenly distributed. Which partitioner behavior should you rely on?

hard

A. Default partitioner: keyed messages to same partition, unkeyed round-robin

B. Custom partitioner that ignores keys and uses random partition

C. Round-robin partitioner for all messages

D. Partitioner that sends all messages to partition 0

Solution

1. Step 1: Understand keyed message behavior

   Default partitioner sends messages with the same key to the same partition.

2. Step 2: Understand unkeyed message behavior

   Messages without keys are distributed evenly using round-robin by default partitioner.

3. Step 3: Match behavior to requirement

   Default partitioner: keyed messages to same partition, unkeyed round-robin matches the requirement exactly.

4. Final Answer:

   Default partitioner: keyed messages to same partition, unkeyed round-robin -> Option A

5. Quick Check:

   Default partitioner fits keyed + unkeyed needs [OK]

Hint: Default partitioner handles keyed and unkeyed well [OK]

Common Mistakes:

• Choosing custom partitioner ignoring keys
• Using round-robin for keyed messages
• Sending all messages to one partition