How to Use Key for Partitioning in Kafka: Simple Guide

In Kafka, you use the key of a message to determine which partition it goes to. Kafka applies a hash function on the key to assign the message to a specific partition, ensuring all messages with the same key go to the same partition.

Syntax

The key is provided when producing a message to Kafka. It is used by Kafka's partitioner to decide the partition number.

Basic syntax for sending a message with a key:

• producer.send(new ProducerRecord(topic, key, value));

Here:

• topic is the Kafka topic name.
• key is the message key used for partitioning.
• value is the message content.

java

producer.send(new ProducerRecord("my-topic", "user123", "message content"));

Example

This example shows a simple Kafka producer in Java sending messages with keys. Messages with the same key go to the same partition, preserving order for that key.

java

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;
import java.util.concurrent.Future;

public class KeyPartitionExample {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        String topic = "my-topic";

        // Send messages with keys
        for (int i = 0; i < 5; i++) {
            String key = "user" + (i % 2); // keys: user0, user1
            String value = "message-" + i;
            ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
            Future<RecordMetadata> future = producer.send(record);
            RecordMetadata metadata = future.get();
            System.out.println("Sent to partition " + metadata.partition() + ": key=" + key + ", value=" + value);
        }

        producer.close();
    }
}

Output

Sent to partition 1: key=user0, value=message-0
Sent to partition 0: key=user1, value=message-1
Sent to partition 1: key=user0, value=message-2
Sent to partition 0: key=user1, value=message-3
Sent to partition 1: key=user0, value=message-4

Common Pitfalls

Not providing a key: If you send messages without a key, Kafka uses round-robin partitioning, so messages with the same logical key may go to different partitions, losing ordering guarantees.

Using a non-serializable or inconsistent key: The key must be serialized properly and consistently to ensure correct partitioning.

Assuming keys guarantee global ordering: Kafka only guarantees order within a partition, so keys only preserve order per key, not across all keys.

java

/* Wrong: No key, messages go to random partitions */
producer.send(new ProducerRecord("my-topic", null, "value1"));
producer.send(new ProducerRecord("my-topic", null, "value2"));

/* Right: Use key to ensure same partition */
producer.send(new ProducerRecord("my-topic", "user123", "value1"));
producer.send(new ProducerRecord("my-topic", "user123", "value2"));

Quick Reference

Concept	Description
Key	Used to determine the partition via hashing
Partition	A subset of topic data; messages with same key go to same partition
ProducerRecord(topic, key, value)	Kafka message with key for partitioning
No key	Messages distributed round-robin, no ordering guarantee per key
Ordering	Guaranteed only within the same partition (same key)

Key Takeaways

Always provide a key when producing messages to control partitioning and preserve order per key.

Kafka hashes the key to assign messages to partitions consistently.

Messages without keys are distributed round-robin, losing key-based ordering.

Keys must be serialized properly to ensure correct partition assignment.

Ordering is guaranteed only within a partition, so keys help maintain order per key.