What is Serializer in Kafka: Explanation and Example

In Kafka, a serializer converts data from an object or data structure into bytes so Kafka can send it over the network. It ensures that messages are transformed into a format Kafka understands before sending them to topics.

How It Works

A serializer in Kafka acts like a translator that changes your data into a language Kafka can send and store. Imagine you want to send a letter to a friend who only understands Morse code. You need to convert your words into dots and dashes first. Similarly, Kafka serializers convert data objects into byte arrays.

This process happens before the data is sent to Kafka topics. When a producer sends a message, the serializer takes the original data (like a string, number, or custom object) and turns it into bytes. On the receiving side, a deserializer converts those bytes back into the original data format.

This mechanism allows Kafka to handle many data types efficiently and ensures messages are consistent and readable by consumers.

Example

This example shows how to use Kafka's built-in StringSerializer to send a simple text message.

java

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class SimpleProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "Hello Kafka");

        producer.send(record);
        producer.close();
    }
}

Output

Message with key 'key1' and value 'Hello Kafka' is sent to topic 'my-topic'.

When to Use

Use serializers whenever you send data to Kafka topics. They are essential for converting your data into bytes so Kafka can transmit it. For example:

• Sending simple text messages with StringSerializer.
• Sending numbers or custom objects using specific serializers like IntegerSerializer or custom ones you write.
• When you want to ensure data consistency and compatibility between producers and consumers.

Without serializers, Kafka cannot understand or transmit your data correctly.

Key Points

• A serializer converts data into bytes for Kafka transmission.
• Kafka provides built-in serializers for common data types like strings and integers.
• You can create custom serializers for complex data formats.
• Serializers work with producers; deserializers work with consumers.
• Using the right serializer ensures data integrity and compatibility.

Key Takeaways

A serializer converts data into bytes so Kafka can send it over the network.

Kafka includes built-in serializers like StringSerializer for common data types.

You must configure serializers in the producer to send messages correctly.

Custom serializers can handle complex or special data formats.

Serializers and deserializers ensure data consistency between producers and consumers.