Deserialization in Kafka - Step-by-Step Execution

Process Flow - Deserialization

Receive byte stream from Kafka topic

↓

Apply deserializer to byte stream

↓

Convert bytes to object/data structure

↓

Pass object to consumer application

↓

Use object in program logic

Deserialization converts raw bytes from Kafka into usable objects for the consumer application.

Execution Sample

Kafka

byte[] data = consumerRecord.value();
MyObject obj = deserializer.deserialize(topic, data);
process(obj);

This code takes bytes from Kafka, deserializes them into an object, then processes it.

Process Table

Step	Action	Input	Output	Notes
1	Receive message	Raw bytes from Kafka topic	Raw bytes	Message arrives as bytes
2	Call deserializer.deserialize()	Raw bytes	MyObject instance	Bytes converted to object
3	Pass object to process()	MyObject instance	Processed result	Object used in app logic
4	End	No more messages	Stop processing	No more data to deserialize

💡 No more messages in Kafka topic, so deserialization stops.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
data	null	Raw bytes	Raw bytes	Raw bytes	Raw bytes
obj	null	null	MyObject instance	MyObject instance	MyObject instance
processedResult	null	null	null	Processed result	Processed result

Key Moments - 2 Insights

Why do we need to convert bytes to an object?

Because Kafka sends data as bytes, and the consumer needs objects to work with. See execution_table step 2 where deserializer converts bytes to an object.

What happens if deserialization fails?

If deserialization fails, the consumer cannot get a usable object, so processing stops or errors. This is before step 3 in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output of step 2?

A Raw bytes

B Processed result

C MyObject instance

D No output

💡 Hint

Check the 'Output' column for step 2 in the execution_table.

At which step does the consumer application start using the deserialized object?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Look at the 'Action' column describing when the object is passed to process() in the execution_table.

If the Kafka topic has no messages, what happens according to the execution_table?

A Deserialization continues endlessly

B Processing stops at step 4

C An error occurs at step 2

D The consumer creates empty objects

💡 Hint

See the 'exit_note' and step 4 in the execution_table.

Concept Snapshot

Deserialization in Kafka:
- Converts byte stream from Kafka into objects
- Uses a deserializer class/method
- Enables consumer to work with data
- Happens before processing logic
- Stops when no more messages

Full Transcript

Deserialization is the process where Kafka consumer takes raw bytes from a Kafka topic and converts them into usable objects. First, the consumer receives the raw byte stream. Then, it calls the deserializer to convert these bytes into an object. After that, the object is passed to the application logic for processing. If there are no more messages, the process stops. This flow ensures that the consumer can work with meaningful data instead of raw bytes.

Practice

1.

What is the main purpose of deserialization in Kafka consumers?

easy

A. To send messages to Kafka topics

B. To compress data before sending to Kafka

C. To convert raw bytes from Kafka into usable data formats

D. To create Kafka topics automatically

Solution

1. Step 1: Understand Kafka message flow

   Kafka stores messages as raw bytes in topics.

2. Step 2: Role of deserialization in consumers

   Deserialization converts these raw bytes back into readable data for the consumer application.

3. Final Answer:

   To convert raw bytes from Kafka into usable data formats -> Option C

4. Quick Check:

   Deserialization = Convert bytes to data [OK]

Hint: Deserialization means reading bytes as data [OK]

Common Mistakes:

• Confusing deserialization with serialization
• Thinking deserialization sends data
• Assuming deserialization compresses data

2.

Which of the following is the correct way to specify a String deserializer in a Kafka consumer configuration?

props.put("key.deserializer", ?);

easy

A. "org.apache.kafka.common.serialization.StringDeserializer"

B. "org.apache.kafka.common.serialization.IntegerSerializer"

C. "org.apache.kafka.common.serialization.ByteArrayDeserializer"

D. "org.apache.kafka.common.serialization.StringSerializer"

Solution

1. Step 1: Identify deserializer for String keys

   The correct class for String deserialization is StringDeserializer.

2. Step 2: Check option correctness

   "org.apache.kafka.common.serialization.StringDeserializer" uses StringDeserializer, which matches the key.deserializer property.

3. Final Answer:

   "org.apache.kafka.common.serialization.StringDeserializer" -> Option A

4. Quick Check:

   String deserializer = StringDeserializer [OK]

Hint: Deserializer ends with 'Deserializer', not 'Serializer' [OK]

Common Mistakes:

• Using Serializer instead of Deserializer
• Mixing IntegerSerializer with deserializer property
• Choosing ByteArrayDeserializer for String data

3.

Given the following Kafka consumer code snippet, what will be the output if the message value is a JSON string but the deserializer is set to StringDeserializer?

props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
ConsumerRecord<String, String> record = consumer.poll(Duration.ofMillis(100)).iterator().next();
System.out.println(record.value());

medium

A. Binary data printed as garbage characters

B. The JSON string as plain text

C. Null value printed

D. A deserialization error exception

Solution

1. Step 1: Understand StringDeserializer behavior

   StringDeserializer converts bytes to a UTF-8 string without parsing JSON.

2. Step 2: Output of printing record.value()

   The JSON string is printed as plain text exactly as received.

3. Final Answer:

   The JSON string as plain text -> Option B

4. Quick Check:

   StringDeserializer outputs raw string [OK]

Hint: StringDeserializer prints raw string, no JSON parsing [OK]

Common Mistakes:

• Expecting automatic JSON parsing
• Thinking deserialization fails on JSON strings
• Assuming binary garbage output

4.

What is the likely cause of this error in a Kafka consumer?

org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition

Given the consumer uses IntegerDeserializer but the producer sends String keys.

medium

A. Mismatch between producer's String keys and consumer's IntegerDeserializer

B. Kafka topic does not exist

C. Consumer group ID is missing

D. Network connection failure

Solution

1. Step 1: Analyze deserialization error cause

   The error occurs when deserializer cannot convert bytes to expected type.

2. Step 2: Check key type mismatch

   Producer sends String keys, but consumer expects Integer keys, causing failure.

3. Final Answer:

   Mismatch between producer's String keys and consumer's IntegerDeserializer -> Option A

4. Quick Check:

   Deserializer mismatch = SerializationException [OK]

Hint: Producer and consumer deserializers must match data type [OK]

Common Mistakes:

• Ignoring type mismatch between producer and consumer
• Blaming unrelated network or topic errors
• Confusing serialization with deserialization errors

5.

You want to consume messages where the value is a JSON object representing a user with fields id (int) and name (string). Which deserialization approach is best to convert the Kafka message value into a User Java object?

hard

A. Use ByteArrayDeserializer and manually convert bytes to User

B. Use StringDeserializer and cast the string to User

C. Use IntegerDeserializer for the value

D. Use a custom deserializer that parses JSON into a User object

Solution

1. Step 1: Understand data format and target type

   The message value is JSON, and we want a User Java object.

2. Step 2: Choose deserialization method

   A custom deserializer can parse JSON and create User instances directly.

3. Step 3: Evaluate other options

   StringDeserializer returns raw JSON string, casting won't convert to User. IntegerDeserializer is wrong type. ByteArrayDeserializer requires manual parsing.

4. Final Answer:

   Use a custom deserializer that parses JSON into a User object -> Option D

5. Quick Check:

   Custom deserializer = JSON to object [OK]

Hint: Custom deserializer parses JSON to Java object directly [OK]

Common Mistakes:

• Casting String to object without parsing
• Using wrong deserializer type
• Manually parsing bytes instead of using deserializer