What is Key in Kafka Message: Explanation and Usage

In Kafka, the key in a message is an optional identifier used to determine the partition where the message is stored. It helps Kafka group related messages together and maintain order within partitions by routing messages with the same key to the same partition.

How It Works

Think of Kafka as a post office with many mailboxes (partitions). The key in a Kafka message acts like the recipient's address. Kafka uses this key to decide which mailbox the message should go to. If two messages have the same key, they will be placed in the same mailbox, keeping them together.

This grouping is important because Kafka guarantees that messages in the same partition are delivered in order. Without a key, messages are distributed randomly across partitions, which can make ordering harder to manage.

Example

This example shows how to send a Kafka message with a key using the Java Kafka client. The key is a string that Kafka uses to decide the partition.

java

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;

public class KafkaKeyExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        String topic = "my-topic";
        String key = "user123";
        String value = "This is a message with a key.";

        ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
        producer.send(record);
        producer.close();

        System.out.println("Message sent with key: " + key);
    }
}

Output

Message sent with key: user123

When to Use

Use a key when you want related messages to be processed in order or grouped together. For example, if you have user events, using the user ID as the key ensures all events for that user go to the same partition and are processed in sequence.

This is useful in scenarios like tracking user activity, processing transactions, or maintaining state per entity. Without keys, messages might be spread randomly, making it harder to keep related data consistent.

Key Points

• The key determines the partition for the message.
• Messages with the same key go to the same partition, preserving order.
• Keys help group related messages for consistent processing.
• Keys are optional but important for ordered or grouped data.

Key Takeaways

The key in a Kafka message decides which partition the message goes to.

Messages with the same key are stored in the same partition to keep order.

Use keys to group related messages for consistent processing.

Keys are optional but essential for ordered or grouped data scenarios.