Partitioner behavior in Kafka - Commands & Configuration

Introduction

Kafka splits data into parts called partitions to spread the load and keep things fast. The partitioner decides which partition each message goes to, helping balance work and keep related messages together.

When you want to send messages about the same user to the same partition to keep their data in order.

When you need to spread messages evenly across partitions to avoid overloading one server.

When you want to control message flow by choosing a custom way to assign partitions.

When troubleshooting message order or load balancing issues in Kafka topics.

When configuring producers to optimize performance and data organization.

Commands

Start a producer that sends messages with keys separated by ':' so Kafka can use the key for partitioning.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.separator=:

Expected Output

>

→

--property parse.key=true - Tells the producer to treat part of the message as the key.

→

--property key.separator=: - Defines ':' as the separator between key and message.

Read all messages from the topic showing which partition and key each message came from to see partitioner behavior.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --property print.partition=true --property print.key=true

Expected Output

partition:0	key:user1	message1

partition:1	key:user2	message2

partition:0	key:user1	message3

→

--property print.partition=true - Shows the partition number for each message.

→

--property print.key=true - Shows the key used for partitioning.

Key Concept

If you remember nothing else from this pattern, remember: the partitioner uses message keys to decide which partition each message goes to, keeping related data together and balancing load.

Common Mistakes

Not setting a key when producing messages.

Without keys, Kafka assigns partitions randomly, which can break message order for related data.

Always provide a key if message order or grouping matters.

Using different key separators than the consumer expects.

If the producer and consumer use different separators, keys won't be parsed correctly, causing wrong partitioning or display.

Use the same key separator in both producer and consumer commands.

Summary

Kafka uses a partitioner to decide which partition each message goes to based on its key.

Setting keys in messages helps keep related data in the same partition and maintain order.

Using console producer and consumer with key and partition properties helps observe partitioner behavior.