What is Partitioner in Kafka: Explanation and Example

A partitioner in Kafka decides which partition a message should go to within a topic. It helps distribute messages evenly or based on keys to ensure order and scalability.

How It Works

Imagine a post office sorting letters into different mailboxes. Each mailbox is like a Kafka partition. The partitioner is the sorter deciding which mailbox each letter goes into. This decision can be random, based on the letter's address (key), or follow a custom rule.

In Kafka, when a producer sends a message, the partitioner uses the message key or other logic to pick a partition. This helps balance load and keeps messages with the same key in order by sending them to the same partition.

Example

This example shows how to use a custom partitioner in Kafka producer configuration in Java. The custom partitioner sends all messages with keys starting with 'A' to partition 0, and others to partition 1.

java

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import java.util.Map;
import java.util.Properties;

public class CustomPartitioner implements Partitioner {
    @Override
    public void configure(Map<String, ?> configs) {}

    @Override
    public int partition(String topic, Object keyObj, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        String key = (String) keyObj;
        if (key != null && key.startsWith("A")) {
            return 0; // send to partition 0
        } else {
            return 1; // send to partition 1
        }
    }

    @Override
    public void close() {}
}

// Producer config snippet
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("partitioner.class", "CustomPartitioner");

Output

Messages with keys starting with 'A' go to partition 0; others go to partition 1.

When to Use

Use a partitioner when you want control over how messages are distributed across partitions. This is important to keep related messages together for ordering or to balance load evenly.

For example, if you have user data keyed by user ID, a partitioner can ensure all messages for the same user go to the same partition. Or, you might want to send high-priority messages to a specific partition for faster processing.

Key Points

• A partitioner decides which partition a Kafka message goes to.
• It can use message keys or custom logic to route messages.
• Helps maintain message order for keys and balance load.
• Kafka provides a default partitioner, but you can create custom ones.

Key Takeaways

A partitioner controls message distribution across Kafka partitions.

It ensures messages with the same key go to the same partition for order.

Custom partitioners let you apply specific routing logic.

Use partitioners to balance load and maintain message order.

Kafka’s default partitioner uses hashing on the key if provided.