Custom SerDes in Kafka - Commands & Configuration

Introduction

When sending data through Kafka, it needs to be turned into bytes and back again. Custom SerDes let you control exactly how your data is changed to bytes and back, so you can use your own formats or special data types.

When you want to send complex objects like JSON or Avro with your own rules.

When the default Kafka serializers don't match your data format needs.

When you want to optimize data size or speed by customizing serialization.

When you need to ensure compatibility between different versions of your data.

When integrating Kafka with systems that require specific data encoding.

Config File - CustomSerdesExample.java

CustomSerdesExample.java

import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serializer;
import java.nio.charset.StandardCharsets;

public class CustomSerdesExample implements Serializer<String>, Deserializer<String> {

    @Override
    public byte[] serialize(String topic, String data) {
        if (data == null) return null;
        return data.toUpperCase().getBytes(StandardCharsets.UTF_8);
    }

    @Override
    public String deserialize(String topic, byte[] data) {
        if (data == null) return null;
        return new String(data, StandardCharsets.UTF_8).toLowerCase();
    }

    @Override
    public void close() {}

    @Override
    public void configure(java.util.Map<String, ?> configs, boolean isKey) {}
}

This Java class defines a custom serializer and deserializer for Kafka that changes strings to uppercase bytes when sending and back to lowercase strings when receiving.

serialize: converts the string to uppercase bytes.

deserialize: converts bytes back to lowercase string.

This example shows how you can control exactly how data is turned into bytes and back.

Commands

Compile the custom serializer/deserializer Java class with Kafka client library in the classpath.

Terminal

javac -cp kafka-clients-3.5.1.jar CustomSerdesExample.java

Expected Output

No output (command runs silently)

Start a Kafka producer using the custom serializer for message values to send data to 'example-topic'.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic --property parse.key=true --property key.serializer=org.apache.kafka.common.serialization.StringSerializer --property value.serializer=CustomSerdesExample

Expected Output

No output (command runs silently)

→

--property value.serializer=CustomSerdesExample - Use the custom serializer for message values

Start a Kafka consumer using the custom deserializer for message values to read data from 'example-topic'.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=CustomSerdesExample

Expected Output

hello

world

→

--property value.deserializer=CustomSerdesExample - Use the custom deserializer for message values

Key Concept

If you remember nothing else from this pattern, remember: custom SerDes let you control exactly how Kafka turns your data into bytes and back, so you can use any format you want.

Common Mistakes

Not including the custom SerDes class in the classpath when running producer or consumer.

Kafka won't find your custom serializer/deserializer and will fail to start or throw errors.

Always compile your custom SerDes and include its jar or class files in the classpath when running Kafka clients.

Using mismatched serializer and deserializer classes between producer and consumer.

Data will not deserialize correctly, causing errors or corrupted data.

Ensure the producer and consumer use matching custom SerDes classes.

Not handling null values in serialize or deserialize methods.

Null pointer exceptions can crash your Kafka clients.

Always check for null and handle it gracefully in your custom SerDes methods.

Summary

Write a Java class implementing Kafka Serializer and Deserializer interfaces to define custom data conversion.

Compile the class and include it in the Kafka client classpath when running producers and consumers.

Use the --property flags to specify your custom serializer and deserializer classes for Kafka clients.

Practice

1.

What is the main purpose of a custom SerDes in Kafka?

easy

A. To manage Kafka topic partitions

B. To speed up Kafka brokers

C. To control how data is converted to bytes and back

D. To monitor Kafka consumer lag

Solution

1. Step 1: Understand SerDes role

   SerDes stands for serializer-deserializer, which converts data to bytes and back.

2. Step 2: Identify main purpose

   Custom SerDes lets you control this conversion for special or complex data.

3. Final Answer:

   To control how data is converted to bytes and back -> Option C

4. Quick Check:

   Custom SerDes = Data conversion control [OK]

Hint: SerDes = Serializer + Deserializer for data [OK]

Common Mistakes:

• Confusing SerDes with Kafka broker settings
• Thinking SerDes manages partitions
• Assuming SerDes monitors consumer lag

2.

Which interface must you implement to create a custom serializer in Kafka?

easy

A. Serializer<T>

B. Consumer<T>

C. Producer<T>

D. Deserializer<T>

Solution

1. Step 1: Recall serializer interface

   Kafka requires implementing Serializer<T> to convert objects to bytes.

2. Step 2: Match interface to purpose

   Deserializer<T> is for reading bytes back, not serialization.

3. Final Answer:

   Serializer<T> -> Option A

4. Quick Check:

   Serializer interface = Serializer<T> [OK]

Hint: Serializer interface handles object to bytes [OK]

Common Mistakes:

• Choosing Deserializer for serialization
• Confusing Producer/Consumer with SerDes
• Using wrong generic interface

3.

Given this serializer code snippet, what will serialize() return for input "hello"?

public byte[] serialize(String topic, String data) {
    return data == null ? null : data.getBytes(StandardCharsets.UTF_8);
}

medium

A. [104, 101, 108, 108, 111]

B. null

C. ["h", "e", "l", "l", "o"]

D. Throws NullPointerException

Solution

1. Step 1: Understand serialize method

   If data is not null, it returns UTF-8 bytes of the string.

2. Step 2: Convert "hello" to bytes

   "hello" in UTF-8 bytes is [104, 101, 108, 108, 111].

3. Final Answer:

   [104, 101, 108, 108, 111] -> Option A

4. Quick Check:

   "hello" bytes = [104,101,108,108,111] [OK]

Hint: String.getBytes(UTF-8) returns byte array [OK]

Common Mistakes:

• Thinking it returns characters array
• Assuming null is returned for non-null input
• Expecting exception on valid input

4.

Identify the error in this deserializer code snippet:

public String deserialize(String topic, byte[] data) {
    return new String(data);
}

medium

A. No error, code is correct

B. Method should return byte[] not String

C. Parameter 'topic' is unused

D. Missing charset in String constructor

Solution

1. Step 1: Check String constructor usage

   Using new String(byte[]) without charset uses platform default, which can cause bugs.

2. Step 2: Identify best practice

   Always specify charset like StandardCharsets.UTF_8 for consistent decoding.

3. Final Answer:

   Missing charset in String constructor -> Option D

4. Quick Check:

   Specify charset in String(byte[]) [OK]

Hint: Always specify charset when converting bytes to String [OK]

Common Mistakes:

• Ignoring charset causes wrong decoding
• Thinking unused parameter is an error
• Assuming return type must be byte[]

5.

You want to send a complex object Person{name, age} through Kafka. Which approach best uses custom SerDes?

1. Create a PersonSerializer that converts Person to JSON bytes.
2. Create a PersonDeserializer that converts JSON bytes back to Person.
3. Register these SerDes in Kafka producer and consumer configs.

What is the main benefit of this approach?

hard

A. It speeds up Kafka message delivery

B. It allows sending complex objects as bytes safely and readably

C. It automatically partitions topics by age

D. It removes the need for Kafka brokers

Solution

1. Step 1: Understand custom SerDes for complex objects

   Serializing to JSON bytes lets complex objects be sent as byte arrays.

2. Step 2: Recognize benefit of JSON format

   JSON is readable and widely supported, making data safe and easy to deserialize.

3. Step 3: Confirm Kafka config usage

   Registering SerDes ensures Kafka knows how to convert data on send and receive.

4. Final Answer:

   It allows sending complex objects as bytes safely and readably -> Option B

5. Quick Check:

   Custom SerDes + JSON = safe complex data transfer [OK]

Hint: Use JSON SerDes for complex object safety [OK]

Common Mistakes:

• Thinking SerDes speeds up Kafka internals
• Assuming SerDes controls topic partitions
• Believing Kafka brokers are not needed