Custom SerDes in Kafka - Mini Project: Build & Apply

Custom SerDes in Kafka

📖 Scenario: You are working on a Kafka application that sends and receives messages containing simple user data. To handle this data properly, you need to create a custom serializer and deserializer (SerDes) that convert user objects to bytes and back.

🎯 Goal: Build a custom SerDes for Kafka that serializes and deserializes a User object with name and age fields.

📋 What You'll Learn

Create a User class with name and age fields

Create a custom serializer class called UserSerializer

Create a custom deserializer class called UserDeserializer

Create a UserSerDe class that implements Kafka's Serializer and Deserializer interfaces

Test the serialization and deserialization by converting a User object to bytes and back

💡 Why This Matters

🌍 Real World

Custom SerDes are used in Kafka applications to efficiently convert complex objects to bytes for transmission and back to objects for processing.

💼 Career

Understanding how to create custom SerDes is important for developers working with Kafka to handle domain-specific data formats and optimize performance.

1

Create the User class

Create a User class with two fields: name of type String and age of type int. Include a constructor to set these fields and override toString() to display the user details.

Kafka

// Your code here

Need a hint?

Define a class with private fields, a constructor, getters, and override toString() to show name and age.

2

Create the UserSerializer class

Create a class called UserSerializer that implements Kafka's Serializer<User> interface. Implement the serialize method to convert a User object into a byte array by encoding the name as UTF-8 bytes, followed by the age as 4 bytes in big-endian order.

Kafka

public class UserSerializer implements org.apache.kafka.common.serialization.Serializer {
    // Your code here
}

Need a hint?

Use ByteBuffer to combine the length of the name, the name bytes, and the age as bytes.

3

Create the UserDeserializer class

Create a class called UserDeserializer that implements Kafka's Deserializer<User> interface. Implement the deserialize method to read the byte array, extract the length of the name, read the name bytes as UTF-8 string, and then read the age as an integer to create a new User object.

Kafka

import org.apache.kafka.common.serialization.Deserializer;

public class UserDeserializer implements Deserializer {
    // Your code here
}

Need a hint?

Use ByteBuffer to read the name length, then the name bytes, then the age integer.

4

Test the UserSerDe serialization and deserialization

Create a User object with name "Alice" and age 30. Use UserSerializer to serialize it to bytes, then use UserDeserializer to deserialize it back to a User object. Finally, print the deserialized User object using System.out.println.

Kafka

public class TestUserSerDe {
    public static void main(String[] args) {
        // Your code here
    }
}

Need a hint?

Create a User object, serialize it, then deserialize it, and print the result.