Partition key and routing in Kafka - Commands & Configuration

Introduction

When sending messages to Kafka, you want related messages to go to the same place so they can be processed in order. Partition keys help decide which partition a message goes to, making sure messages with the same key stay together.

When you want all messages about the same user to be processed in order.

When you need to balance message load evenly across servers.

When you want to group messages by a category like product ID or region.

When you want to make sure related events are handled by the same consumer.

When you want to improve performance by controlling message distribution.

Config File - producer.properties

producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

This file configures the Kafka producer client.

bootstrap.servers tells the producer where the Kafka servers are.

key.serializer and value.serializer tell Kafka how to turn your keys and messages into bytes.

Commands

This command starts a Kafka producer that reads messages from your keyboard. It uses the text before ':' as the partition key to decide where to send each message.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

--property parse.key=true - Enable parsing of keys from input lines

→

--property key.separator=: - Set ':' as the separator between key and message

This sends a single message with key 'user1' and value 'Hello Kafka' to the topic. The key ensures this message goes to the same partition as other 'user1' messages.

Terminal

echo "user1:Hello Kafka" | kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.separator=:

Expected Output

No output (command runs silently)

→

--property parse.key=true - Enable parsing of keys from input lines

→

--property key.separator=: - Set ':' as the separator between key and message

This command reads messages from the topic from the start and shows both the key and the message, so you can see how keys are used for routing.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --property print.key=true --property key.separator=:

Expected Output

user1:Hello Kafka

→

--property print.key=true - Show the key along with the message

→

--property key.separator=: - Set ':' as the separator between key and message

Key Concept

Partition keys decide which partition a message goes to, keeping related messages together for ordered processing.

Common Mistakes

Not setting parse.key=true when sending messages with keys

Kafka treats the whole message as value and ignores the key, so messages won't be routed as expected.

Always use --property parse.key=true and set the key separator when sending keyed messages.

Using different key separators between producer and consumer

The consumer won't correctly display keys if the separator doesn't match the producer's setting.

Use the same key.separator value for both producer and consumer commands.

Sending messages without keys when order matters for a group

Messages may go to different partitions and be processed out of order.

Always provide a partition key for messages that need ordered processing together.

Summary

Use partition keys to control which partition a Kafka message goes to.

Set --property parse.key=true and key.separator when producing messages with keys.

Use --property print.key=true and matching key.separator when consuming to see keys.