Deserialization in Kafka - Commands & Configuration

Introduction

Deserialization is the process of converting data received from Kafka into a format your application can understand. It solves the problem of reading raw bytes from Kafka messages and turning them into usable objects.

When your application reads messages from Kafka topics and needs to convert them into objects or data structures.

When you want to process JSON or Avro formatted messages from Kafka in your consumer application.

When you need to ensure the data format matches what your application expects to avoid errors.

When integrating Kafka with systems that require specific data formats like strings, JSON, or custom objects.

When debugging message consumption issues caused by incorrect data formats or serialization mismatches.

Config File - consumer.properties

consumer.properties

bootstrap.servers=localhost:9092
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
group.id=my-consumer-group
enable.auto.commit=true
auto.commit.interval.ms=1000

This configuration file sets up a Kafka consumer.

bootstrap.servers tells the consumer where the Kafka server is.

key.deserializer and value.deserializer specify how to convert the message key and value from bytes to strings.

group.id identifies the consumer group.

enable.auto.commit and auto.commit.interval.ms control automatic offset commits.

Commands

This command starts a Kafka consumer that reads messages from 'example-topic' from the beginning. It uses StringDeserializer to convert message keys and values from bytes to readable strings.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning --property print.key=true --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer --property value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

Expected Output

key1: Hello World

key2: Kafka is fun

key3: Deserialization example

→

--bootstrap-server - Specifies the Kafka server address

→

--topic - Specifies the Kafka topic to consume from

→

--property key.deserializer - Sets the deserializer for message keys

This command starts a Kafka producer to send messages to 'example-topic'. It helps test deserialization by sending known messages.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka server address

→

--topic - Specifies the Kafka topic to send messages to

Key Concept

If you remember nothing else from this pattern, remember: deserialization converts raw Kafka message bytes into usable data formats your application can work with.

Common Mistakes

Using the wrong deserializer class for the message format.

This causes errors or unreadable data because the bytes are interpreted incorrectly.

Always match the deserializer to the message format, like StringDeserializer for strings or AvroDeserializer for Avro data.

Not specifying deserializer properties when running Kafka console consumer.

Kafka defaults to byte arrays, so output will be unreadable raw bytes.

Use --property key.deserializer and value.deserializer flags to specify the correct deserializers.

Summary

Configure your Kafka consumer with the correct deserializer classes to convert message bytes into usable data.

Use kafka-console-consumer with deserializer properties to read and verify messages in a readable format.

Test deserialization by sending known messages with kafka-console-producer and consuming them.

Practice

1.

What is the main purpose of deserialization in Kafka consumers?

easy

A. To send messages to Kafka topics

B. To compress data before sending to Kafka

C. To convert raw bytes from Kafka into usable data formats

D. To create Kafka topics automatically

Solution

1. Step 1: Understand Kafka message flow

   Kafka stores messages as raw bytes in topics.

2. Step 2: Role of deserialization in consumers

   Deserialization converts these raw bytes back into readable data for the consumer application.

3. Final Answer:

   To convert raw bytes from Kafka into usable data formats -> Option C

4. Quick Check:

   Deserialization = Convert bytes to data [OK]

Hint: Deserialization means reading bytes as data [OK]

Common Mistakes:

• Confusing deserialization with serialization
• Thinking deserialization sends data
• Assuming deserialization compresses data

2.

Which of the following is the correct way to specify a String deserializer in a Kafka consumer configuration?

props.put("key.deserializer", ?);

easy

A. "org.apache.kafka.common.serialization.StringDeserializer"

B. "org.apache.kafka.common.serialization.IntegerSerializer"

C. "org.apache.kafka.common.serialization.ByteArrayDeserializer"

D. "org.apache.kafka.common.serialization.StringSerializer"

Solution

1. Step 1: Identify deserializer for String keys

   The correct class for String deserialization is StringDeserializer.

2. Step 2: Check option correctness

   "org.apache.kafka.common.serialization.StringDeserializer" uses StringDeserializer, which matches the key.deserializer property.

3. Final Answer:

   "org.apache.kafka.common.serialization.StringDeserializer" -> Option A

4. Quick Check:

   String deserializer = StringDeserializer [OK]

Hint: Deserializer ends with 'Deserializer', not 'Serializer' [OK]

Common Mistakes:

• Using Serializer instead of Deserializer
• Mixing IntegerSerializer with deserializer property
• Choosing ByteArrayDeserializer for String data

3.

Given the following Kafka consumer code snippet, what will be the output if the message value is a JSON string but the deserializer is set to StringDeserializer?

props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
ConsumerRecord<String, String> record = consumer.poll(Duration.ofMillis(100)).iterator().next();
System.out.println(record.value());

medium

A. Binary data printed as garbage characters

B. The JSON string as plain text

C. Null value printed

D. A deserialization error exception

Solution

1. Step 1: Understand StringDeserializer behavior

   StringDeserializer converts bytes to a UTF-8 string without parsing JSON.

2. Step 2: Output of printing record.value()

   The JSON string is printed as plain text exactly as received.

3. Final Answer:

   The JSON string as plain text -> Option B

4. Quick Check:

   StringDeserializer outputs raw string [OK]

Hint: StringDeserializer prints raw string, no JSON parsing [OK]

Common Mistakes:

• Expecting automatic JSON parsing
• Thinking deserialization fails on JSON strings
• Assuming binary garbage output

4.

What is the likely cause of this error in a Kafka consumer?

org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition

Given the consumer uses IntegerDeserializer but the producer sends String keys.

medium

A. Mismatch between producer's String keys and consumer's IntegerDeserializer

B. Kafka topic does not exist

C. Consumer group ID is missing

D. Network connection failure

Solution

1. Step 1: Analyze deserialization error cause

   The error occurs when deserializer cannot convert bytes to expected type.

2. Step 2: Check key type mismatch

   Producer sends String keys, but consumer expects Integer keys, causing failure.

3. Final Answer:

   Mismatch between producer's String keys and consumer's IntegerDeserializer -> Option A

4. Quick Check:

   Deserializer mismatch = SerializationException [OK]

Hint: Producer and consumer deserializers must match data type [OK]

Common Mistakes:

• Ignoring type mismatch between producer and consumer
• Blaming unrelated network or topic errors
• Confusing serialization with deserialization errors

5.

You want to consume messages where the value is a JSON object representing a user with fields id (int) and name (string). Which deserialization approach is best to convert the Kafka message value into a User Java object?

hard

A. Use ByteArrayDeserializer and manually convert bytes to User

B. Use StringDeserializer and cast the string to User

C. Use IntegerDeserializer for the value

D. Use a custom deserializer that parses JSON into a User object

Solution

1. Step 1: Understand data format and target type

   The message value is JSON, and we want a User Java object.

2. Step 2: Choose deserialization method

   A custom deserializer can parse JSON and create User instances directly.

3. Step 3: Evaluate other options

   StringDeserializer returns raw JSON string, casting won't convert to User. IntegerDeserializer is wrong type. ByteArrayDeserializer requires manual parsing.

4. Final Answer:

   Use a custom deserializer that parses JSON into a User object -> Option D

5. Quick Check:

   Custom deserializer = JSON to object [OK]

Hint: Custom deserializer parses JSON to Java object directly [OK]

Common Mistakes:

• Casting String to object without parsing
• Using wrong deserializer type
• Manually parsing bytes instead of using deserializer