Custom SerDes in Kafka - Mini Project: Build & Apply

Custom SerDes in Kafka

📖 Scenario: You are working on a Kafka application that sends and receives messages containing simple user data. To handle this data properly, you need to create a custom serializer and deserializer (SerDes) that convert user objects to bytes and back.

🎯 Goal: Build a custom SerDes for Kafka that serializes and deserializes a User object with name and age fields.

📋 What You'll Learn

Create a User class with name and age fields

Create a custom serializer class called UserSerializer

Create a custom deserializer class called UserDeserializer

Create a UserSerDe class that implements Kafka's Serializer and Deserializer interfaces

Test the serialization and deserialization by converting a User object to bytes and back

💡 Why This Matters

🌍 Real World

Custom SerDes are used in Kafka applications to efficiently convert complex objects to bytes for transmission and back to objects for processing.

💼 Career

Understanding how to create custom SerDes is important for developers working with Kafka to handle domain-specific data formats and optimize performance.

1

Create the User class

Create a User class with two fields: name of type String and age of type int. Include a constructor to set these fields and override toString() to display the user details.

Kafka

// Your code here

Hint

Define a class with private fields, a constructor, getters, and override toString() to show name and age.

2

Create the UserSerializer class

Create a class called UserSerializer that implements Kafka's Serializer<User> interface. Implement the serialize method to convert a User object into a byte array by encoding the name as UTF-8 bytes, followed by the age as 4 bytes in big-endian order.

Kafka

public class UserSerializer implements org.apache.kafka.common.serialization.Serializer {
    // Your code here
}

Hint

Use ByteBuffer to combine the length of the name, the name bytes, and the age as bytes.

3

Create the UserDeserializer class

Create a class called UserDeserializer that implements Kafka's Deserializer<User> interface. Implement the deserialize method to read the byte array, extract the length of the name, read the name bytes as UTF-8 string, and then read the age as an integer to create a new User object.

Kafka

import org.apache.kafka.common.serialization.Deserializer;

public class UserDeserializer implements Deserializer {
    // Your code here
}

Hint

Use ByteBuffer to read the name length, then the name bytes, then the age integer.

4

Test the UserSerDe serialization and deserialization

Create a User object with name "Alice" and age 30. Use UserSerializer to serialize it to bytes, then use UserDeserializer to deserialize it back to a User object. Finally, print the deserialized User object using System.out.println.

Kafka

public class TestUserSerDe {
    public static void main(String[] args) {
        // Your code here
    }
}

Hint

Create a User object, serialize it, then deserialize it, and print the result.

Practice

1.

What is the main purpose of a custom SerDes in Kafka?

easy

A. To manage Kafka topic partitions

B. To speed up Kafka brokers

C. To control how data is converted to bytes and back

D. To monitor Kafka consumer lag

Solution

1. Step 1: Understand SerDes role

   SerDes stands for serializer-deserializer, which converts data to bytes and back.

2. Step 2: Identify main purpose

   Custom SerDes lets you control this conversion for special or complex data.

3. Final Answer:

   To control how data is converted to bytes and back -> Option C

4. Quick Check:

   Custom SerDes = Data conversion control [OK]

Hint: SerDes = Serializer + Deserializer for data [OK]

Common Mistakes:

• Confusing SerDes with Kafka broker settings
• Thinking SerDes manages partitions
• Assuming SerDes monitors consumer lag

2.

Which interface must you implement to create a custom serializer in Kafka?

easy

A. Serializer<T>

B. Consumer<T>

C. Producer<T>

D. Deserializer<T>

Solution

1. Step 1: Recall serializer interface

   Kafka requires implementing Serializer<T> to convert objects to bytes.

2. Step 2: Match interface to purpose

   Deserializer<T> is for reading bytes back, not serialization.

3. Final Answer:

   Serializer<T> -> Option A

4. Quick Check:

   Serializer interface = Serializer<T> [OK]

Hint: Serializer interface handles object to bytes [OK]

Common Mistakes:

• Choosing Deserializer for serialization
• Confusing Producer/Consumer with SerDes
• Using wrong generic interface

3.

Given this serializer code snippet, what will serialize() return for input "hello"?

public byte[] serialize(String topic, String data) {
    return data == null ? null : data.getBytes(StandardCharsets.UTF_8);
}

medium

A. [104, 101, 108, 108, 111]

B. null

C. ["h", "e", "l", "l", "o"]

D. Throws NullPointerException

Solution

1. Step 1: Understand serialize method

   If data is not null, it returns UTF-8 bytes of the string.

2. Step 2: Convert "hello" to bytes

   "hello" in UTF-8 bytes is [104, 101, 108, 108, 111].

3. Final Answer:

   [104, 101, 108, 108, 111] -> Option A

4. Quick Check:

   "hello" bytes = [104,101,108,108,111] [OK]

Hint: String.getBytes(UTF-8) returns byte array [OK]

Common Mistakes:

• Thinking it returns characters array
• Assuming null is returned for non-null input
• Expecting exception on valid input

4.

Identify the error in this deserializer code snippet:

public String deserialize(String topic, byte[] data) {
    return new String(data);
}

medium

A. No error, code is correct

B. Method should return byte[] not String

C. Parameter 'topic' is unused

D. Missing charset in String constructor

Solution

1. Step 1: Check String constructor usage

   Using new String(byte[]) without charset uses platform default, which can cause bugs.

2. Step 2: Identify best practice

   Always specify charset like StandardCharsets.UTF_8 for consistent decoding.

3. Final Answer:

   Missing charset in String constructor -> Option D

4. Quick Check:

   Specify charset in String(byte[]) [OK]

Hint: Always specify charset when converting bytes to String [OK]

Common Mistakes:

• Ignoring charset causes wrong decoding
• Thinking unused parameter is an error
• Assuming return type must be byte[]

5.

You want to send a complex object Person{name, age} through Kafka. Which approach best uses custom SerDes?

1. Create a PersonSerializer that converts Person to JSON bytes.
2. Create a PersonDeserializer that converts JSON bytes back to Person.
3. Register these SerDes in Kafka producer and consumer configs.

What is the main benefit of this approach?

hard

A. It speeds up Kafka message delivery

B. It allows sending complex objects as bytes safely and readably

C. It automatically partitions topics by age

D. It removes the need for Kafka brokers

Solution

1. Step 1: Understand custom SerDes for complex objects

   Serializing to JSON bytes lets complex objects be sent as byte arrays.

2. Step 2: Recognize benefit of JSON format

   JSON is readable and widely supported, making data safe and easy to deserialize.

3. Step 3: Confirm Kafka config usage

   Registering SerDes ensures Kafka knows how to convert data on send and receive.

4. Final Answer:

   It allows sending complex objects as bytes safely and readably -> Option B

5. Quick Check:

   Custom SerDes + JSON = safe complex data transfer [OK]

Hint: Use JSON SerDes for complex object safety [OK]

Common Mistakes:

• Thinking SerDes speeds up Kafka internals
• Assuming SerDes controls topic partitions
• Believing Kafka brokers are not needed