Custom SerDes in Kafka - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does SerDes stand for in Kafka?

SerDes stands for Serializer/Deserializer. It converts data to bytes for sending and back to objects when receiving.

beginner

Why create a custom SerDes in Kafka?

To handle special data formats or complex objects that default serializers can't process well.

intermediate

What two main methods must a Kafka Serializer implement?

The methods are serialize(String topic, T data) to convert data to bytes, and close() to release resources.

beginner

What is the role of the Deserializer in a Kafka SerDes?

It converts byte arrays back into objects so the consumer can use the data in its original form.

intermediate

How do you register a custom SerDes in Kafka producer or consumer?

You set the key.serializer or value.serializer (for producer) and key.deserializer or value.deserializer (for consumer) properties to your custom Serializer and Deserializer classes respectively.

What is the main purpose of a Kafka Serializer?

A Send messages to Kafka topics

B Convert byte arrays to objects

C Convert objects to byte arrays

D Manage Kafka partitions

A serializer converts objects into byte arrays so Kafka can send them over the network.

Which interface must you implement to create a custom Kafka Deserializer?

A org.apache.kafka.common.serialization.Deserializer

B org.apache.kafka.common.serialization.Serializer

C org.apache.kafka.clients.consumer.Consumer

D org.apache.kafka.clients.producer.Producer

The Deserializer interface is used to convert byte arrays back into objects.

If your data is a complex Java object, what is a good reason to create a custom SerDes?

A Default SerDes may not handle complex objects correctly

B To speed up Kafka brokers

C To avoid using Kafka topics

D To change Kafka partitioning

Custom SerDes allow you to control how complex objects are converted to bytes and back.

Where do you specify your custom SerDes class in Kafka configuration?

A In the consumer group ID

B In the Kafka broker settings

C In the Kafka topic name

D In the serializer and deserializer properties

You set the serializer and deserializer properties to your custom class in producer or consumer configs.

What happens if you don't implement the close() method in your custom SerDes?

A Kafka will not start

B Resources may not be released properly

C Messages will be lost

D Serialization will fail

The close method helps clean up resources like streams or buffers to avoid memory leaks.

Explain how a custom SerDes works in Kafka and why you might need one.

Think about how data travels from your program to Kafka and back.

Describe the steps to create and use a custom SerDes in a Kafka application.

Focus on coding and configuration steps.

Practice

1.

What is the main purpose of a custom SerDes in Kafka?

easy

A. To manage Kafka topic partitions

B. To speed up Kafka brokers

C. To control how data is converted to bytes and back

D. To monitor Kafka consumer lag

Solution

1. Step 1: Understand SerDes role

   SerDes stands for serializer-deserializer, which converts data to bytes and back.

2. Step 2: Identify main purpose

   Custom SerDes lets you control this conversion for special or complex data.

3. Final Answer:

   To control how data is converted to bytes and back -> Option C

4. Quick Check:

   Custom SerDes = Data conversion control [OK]

Hint: SerDes = Serializer + Deserializer for data [OK]

Common Mistakes:

• Confusing SerDes with Kafka broker settings
• Thinking SerDes manages partitions
• Assuming SerDes monitors consumer lag

2.

Which interface must you implement to create a custom serializer in Kafka?

easy

A. Serializer<T>

B. Consumer<T>

C. Producer<T>

D. Deserializer<T>

Solution

1. Step 1: Recall serializer interface

   Kafka requires implementing Serializer<T> to convert objects to bytes.

2. Step 2: Match interface to purpose

   Deserializer<T> is for reading bytes back, not serialization.

3. Final Answer:

   Serializer<T> -> Option A

4. Quick Check:

   Serializer interface = Serializer<T> [OK]

Hint: Serializer interface handles object to bytes [OK]

Common Mistakes:

• Choosing Deserializer for serialization
• Confusing Producer/Consumer with SerDes
• Using wrong generic interface

3.

Given this serializer code snippet, what will serialize() return for input "hello"?

public byte[] serialize(String topic, String data) {
    return data == null ? null : data.getBytes(StandardCharsets.UTF_8);
}

medium

A. [104, 101, 108, 108, 111]

B. null

C. ["h", "e", "l", "l", "o"]

D. Throws NullPointerException

Solution

1. Step 1: Understand serialize method

   If data is not null, it returns UTF-8 bytes of the string.

2. Step 2: Convert "hello" to bytes

   "hello" in UTF-8 bytes is [104, 101, 108, 108, 111].

3. Final Answer:

   [104, 101, 108, 108, 111] -> Option A

4. Quick Check:

   "hello" bytes = [104,101,108,108,111] [OK]

Hint: String.getBytes(UTF-8) returns byte array [OK]

Common Mistakes:

• Thinking it returns characters array
• Assuming null is returned for non-null input
• Expecting exception on valid input

4.

Identify the error in this deserializer code snippet:

public String deserialize(String topic, byte[] data) {
    return new String(data);
}

medium

A. No error, code is correct

B. Method should return byte[] not String

C. Parameter 'topic' is unused

D. Missing charset in String constructor

Solution

1. Step 1: Check String constructor usage

   Using new String(byte[]) without charset uses platform default, which can cause bugs.

2. Step 2: Identify best practice

   Always specify charset like StandardCharsets.UTF_8 for consistent decoding.

3. Final Answer:

   Missing charset in String constructor -> Option D

4. Quick Check:

   Specify charset in String(byte[]) [OK]

Hint: Always specify charset when converting bytes to String [OK]

Common Mistakes:

• Ignoring charset causes wrong decoding
• Thinking unused parameter is an error
• Assuming return type must be byte[]

5.

You want to send a complex object Person{name, age} through Kafka. Which approach best uses custom SerDes?

1. Create a PersonSerializer that converts Person to JSON bytes.
2. Create a PersonDeserializer that converts JSON bytes back to Person.
3. Register these SerDes in Kafka producer and consumer configs.

What is the main benefit of this approach?

hard

A. It speeds up Kafka message delivery

B. It allows sending complex objects as bytes safely and readably

C. It automatically partitions topics by age

D. It removes the need for Kafka brokers

Solution

1. Step 1: Understand custom SerDes for complex objects

   Serializing to JSON bytes lets complex objects be sent as byte arrays.

2. Step 2: Recognize benefit of JSON format

   JSON is readable and widely supported, making data safe and easy to deserialize.

3. Step 3: Confirm Kafka config usage

   Registering SerDes ensures Kafka knows how to convert data on send and receive.

4. Final Answer:

   It allows sending complex objects as bytes safely and readably -> Option B

5. Quick Check:

   Custom SerDes + JSON = safe complex data transfer [OK]

Hint: Use JSON SerDes for complex object safety [OK]

Common Mistakes:

• Thinking SerDes speeds up Kafka internals
• Assuming SerDes controls topic partitions
• Believing Kafka brokers are not needed