Custom SerDes in Kafka - Step-by-Step Execution

Process Flow - Custom SerDes

Start

↓

Serialize Object

↓

Send Bytes to Kafka

↓

Receive Bytes from Kafka

↓

Deserialize Bytes to Object

↓

Use Object in Application

↓

End

This flow shows how a custom serializer converts an object to bytes before sending to Kafka, and a custom deserializer converts bytes back to the object after receiving.

Execution Sample

Kafka

public class UserSerializer implements Serializer<User> {
  @Override
  public byte[] serialize(String topic, User data) {
    return (data.getName() + "," + data.getAge()).getBytes();
  }
}

public class UserDeserializer implements Deserializer<User> {
  @Override
  public User deserialize(String topic, byte[] data) {
    String[] parts = new String(data).split(",");
    return new User(parts[0], Integer.parseInt(parts[1]));
  }
}

This code serializes a User object to bytes by joining name and age with a comma, and deserializes bytes back to a User object by splitting the string.

Process Table

Step	Action	Input	Output	Notes
1	Serialize User object	User(name=Alice, age=30)	byte[] representing "Alice,30"	Convert object fields to string, then to bytes
2	Send bytes to Kafka	byte[] "Alice,30"	Sent to Kafka topic	Bytes are transmitted over Kafka
3	Receive bytes from Kafka	byte[] "Alice,30"	byte[] "Alice,30"	Bytes received unchanged
4	Deserialize bytes	byte[] "Alice,30"	User(name=Alice, age=30)	Split string and parse age to int
5	Use User object	User(name=Alice, age=30)	User object ready	Application can now use the User object
6	End	-	-	Process complete

💡 Deserialization completes successfully when bytes are converted back to the original User object.

Status Tracker

Variable	Start	After Serialize	After Send	After Receive	After Deserialize	Final
user	User(name=Alice, age=30)	User(name=Alice, age=30)	User(name=Alice, age=30)	User(name=Alice, age=30)	User(name=Alice, age=30)	User(name=Alice, age=30)
bytes	null	byte[] "Alice,30"	byte[] "Alice,30"	byte[] "Alice,30"	null	null
deserializedUser	null	null	null	null	User(name=Alice, age=30)	User(name=Alice, age=30)

Key Moments - 3 Insights

Why do we convert the User object to a string before bytes?

Because Kafka sends data as bytes, we must convert the object to a byte array. Converting to a string first makes it easy to join fields and then get bytes, as shown in step 1 of the execution_table.

What happens if the deserializer receives corrupted bytes?

If bytes are corrupted, splitting or parsing may fail or produce wrong data. The deserializer expects the format shown in step 4. If format differs, errors or wrong objects occur.

Why do we need both serializer and deserializer?

Serializer converts objects to bytes for Kafka to send. Deserializer converts bytes back to objects for the application to use. Both must agree on the format, as seen in steps 1 and 4.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of step 1 (Serialize User object)?

A String "Alice,30"

B User object with name and age

C byte[] representing "Alice,30"

D null

💡 Hint

Check the Output column in step 1 of the execution_table.

At which step does the data change from bytes back to a User object?

A Step 4

B Step 2

C Step 3

D Step 5

💡 Hint

Look for the step where deserialize is called in the execution_table.

If the serializer joined fields with a semicolon instead of a comma, what would change in the execution_table?

A Step 5 would produce a different User object

B Step 1 output would have semicolon instead of comma

C Step 4 deserialization would fail without changes

D No changes needed anywhere

💡 Hint

Check the string format in step 1 output and how step 4 splits the string.

Concept Snapshot

Custom SerDes in Kafka:
- Serializer converts objects to byte arrays before sending.
- Deserializer converts byte arrays back to objects after receiving.
- Both must agree on the data format (e.g., CSV string).
- Implement Serializer<T> and Deserializer<T> interfaces.
- Used to send complex objects through Kafka topics.

Full Transcript

This visual execution shows how custom serialization and deserialization work in Kafka. First, a User object is converted to a byte array by joining its fields as a string and getting bytes. These bytes are sent through Kafka. On the receiving side, the bytes are converted back to a string, split by comma, and parsed to recreate the User object. Variables like user, bytes, and deserializedUser change as the program runs. Key moments include why conversion to string is needed, what happens if bytes are corrupted, and why both serializer and deserializer are required. The quizzes test understanding of each step's input and output and the importance of matching formats. This process allows Kafka to handle complex data types by converting them to bytes and back.

Practice

1.

What is the main purpose of a custom SerDes in Kafka?

easy

A. To manage Kafka topic partitions

B. To speed up Kafka brokers

C. To control how data is converted to bytes and back

D. To monitor Kafka consumer lag

Solution

1. Step 1: Understand SerDes role

   SerDes stands for serializer-deserializer, which converts data to bytes and back.

2. Step 2: Identify main purpose

   Custom SerDes lets you control this conversion for special or complex data.

3. Final Answer:

   To control how data is converted to bytes and back -> Option C

4. Quick Check:

   Custom SerDes = Data conversion control [OK]

Hint: SerDes = Serializer + Deserializer for data [OK]

Common Mistakes:

• Confusing SerDes with Kafka broker settings
• Thinking SerDes manages partitions
• Assuming SerDes monitors consumer lag

2.

Which interface must you implement to create a custom serializer in Kafka?

easy

A. Serializer<T>

B. Consumer<T>

C. Producer<T>

D. Deserializer<T>

Solution

1. Step 1: Recall serializer interface

   Kafka requires implementing Serializer<T> to convert objects to bytes.

2. Step 2: Match interface to purpose

   Deserializer<T> is for reading bytes back, not serialization.

3. Final Answer:

   Serializer<T> -> Option A

4. Quick Check:

   Serializer interface = Serializer<T> [OK]

Hint: Serializer interface handles object to bytes [OK]

Common Mistakes:

• Choosing Deserializer for serialization
• Confusing Producer/Consumer with SerDes
• Using wrong generic interface

3.

Given this serializer code snippet, what will serialize() return for input "hello"?

public byte[] serialize(String topic, String data) {
    return data == null ? null : data.getBytes(StandardCharsets.UTF_8);
}

medium

A. [104, 101, 108, 108, 111]

B. null

C. ["h", "e", "l", "l", "o"]

D. Throws NullPointerException

Solution

1. Step 1: Understand serialize method

   If data is not null, it returns UTF-8 bytes of the string.

2. Step 2: Convert "hello" to bytes

   "hello" in UTF-8 bytes is [104, 101, 108, 108, 111].

3. Final Answer:

   [104, 101, 108, 108, 111] -> Option A

4. Quick Check:

   "hello" bytes = [104,101,108,108,111] [OK]

Hint: String.getBytes(UTF-8) returns byte array [OK]

Common Mistakes:

• Thinking it returns characters array
• Assuming null is returned for non-null input
• Expecting exception on valid input

4.

Identify the error in this deserializer code snippet:

public String deserialize(String topic, byte[] data) {
    return new String(data);
}

medium

A. No error, code is correct

B. Method should return byte[] not String

C. Parameter 'topic' is unused

D. Missing charset in String constructor

Solution

1. Step 1: Check String constructor usage

   Using new String(byte[]) without charset uses platform default, which can cause bugs.

2. Step 2: Identify best practice

   Always specify charset like StandardCharsets.UTF_8 for consistent decoding.

3. Final Answer:

   Missing charset in String constructor -> Option D

4. Quick Check:

   Specify charset in String(byte[]) [OK]

Hint: Always specify charset when converting bytes to String [OK]

Common Mistakes:

• Ignoring charset causes wrong decoding
• Thinking unused parameter is an error
• Assuming return type must be byte[]

5.

You want to send a complex object Person{name, age} through Kafka. Which approach best uses custom SerDes?

1. Create a PersonSerializer that converts Person to JSON bytes.
2. Create a PersonDeserializer that converts JSON bytes back to Person.
3. Register these SerDes in Kafka producer and consumer configs.

What is the main benefit of this approach?

hard

A. It speeds up Kafka message delivery

B. It allows sending complex objects as bytes safely and readably

C. It automatically partitions topics by age

D. It removes the need for Kafka brokers

Solution

1. Step 1: Understand custom SerDes for complex objects

   Serializing to JSON bytes lets complex objects be sent as byte arrays.

2. Step 2: Recognize benefit of JSON format

   JSON is readable and widely supported, making data safe and easy to deserialize.

3. Step 3: Confirm Kafka config usage

   Registering SerDes ensures Kafka knows how to convert data on send and receive.

4. Final Answer:

   It allows sending complex objects as bytes safely and readably -> Option B

5. Quick Check:

   Custom SerDes + JSON = safe complex data transfer [OK]

Hint: Use JSON SerDes for complex object safety [OK]

Common Mistakes:

• Thinking SerDes speeds up Kafka internals
• Assuming SerDes controls topic partitions
• Believing Kafka brokers are not needed