Deserialization in Kafka - Mini Project: Build & Apply

Deserialization with Kafka Consumer

📖 Scenario: You are working with Kafka to read messages from a topic. These messages are sent as JSON strings. To use them in your program, you need to convert these JSON strings back into Python dictionaries. This process is called deserialization.

🎯 Goal: Build a simple Kafka consumer that reads JSON messages from a topic and deserializes them into Python dictionaries.

📋 What You'll Learn

Create a Kafka consumer that connects to a local Kafka server

Configure the consumer to read from the topic named test-topic

Deserialize the JSON string messages into Python dictionaries

Print the deserialized dictionary for each message

💡 Why This Matters

🌍 Real World

Kafka is widely used to stream data between systems. Deserialization is essential to convert raw message bytes into usable data structures.

💼 Career

Understanding Kafka deserialization is important for roles in data engineering, backend development, and real-time data processing.

1

Set up Kafka consumer

Import KafkaConsumer from kafka and create a consumer called consumer that connects to localhost:9092 and subscribes to the topic 'test-topic'.

Kafka

# Import KafkaConsumer and create consumer
# Your code here

Hint

Use KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092']) to create the consumer.

2

Add JSON deserialization setup

Import the json module and add a variable called decode_utf8 that decodes bytes to UTF-8 strings. This will help convert message values from bytes to strings before deserialization.

Kafka

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092'])

# Create decode_utf8 function to decode bytes to string
# Your code here

Hint

Define a function decode_utf8 that takes data and returns data.decode('utf-8').

3

Deserialize messages in the consumer loop

Use a for loop with variable message to iterate over consumer. Inside the loop, decode the message value using decode_utf8 and then deserialize it using json.loads into a variable called data.

Kafka

from kafka import KafkaConsumer
import json

def decode_utf8(data):
    return data.decode('utf-8')

consumer = KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092'])

# Loop over consumer and deserialize messages
# Your code here

Hint

Use for message in consumer: and inside decode with decode_utf8(message.value) then deserialize with json.loads.

4

Print the deserialized data

Inside the for loop, add a print statement to display the data dictionary.

Kafka

from kafka import KafkaConsumer
import json

def decode_utf8(data):
    return data.decode('utf-8')

consumer = KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092'])

for message in consumer:
    json_str = decode_utf8(message.value)
    data = json.loads(json_str)
    # Print the deserialized data
    # Your code here

Hint

Use print(data) inside the loop to show the dictionary.

Practice

1.

What is the main purpose of deserialization in Kafka consumers?

easy

A. To send messages to Kafka topics

B. To compress data before sending to Kafka

C. To convert raw bytes from Kafka into usable data formats

D. To create Kafka topics automatically

Solution

1. Step 1: Understand Kafka message flow

   Kafka stores messages as raw bytes in topics.

2. Step 2: Role of deserialization in consumers

   Deserialization converts these raw bytes back into readable data for the consumer application.

3. Final Answer:

   To convert raw bytes from Kafka into usable data formats -> Option C

4. Quick Check:

   Deserialization = Convert bytes to data [OK]

Hint: Deserialization means reading bytes as data [OK]

Common Mistakes:

• Confusing deserialization with serialization
• Thinking deserialization sends data
• Assuming deserialization compresses data

2.

Which of the following is the correct way to specify a String deserializer in a Kafka consumer configuration?

props.put("key.deserializer", ?);

easy

A. "org.apache.kafka.common.serialization.StringDeserializer"

B. "org.apache.kafka.common.serialization.IntegerSerializer"

C. "org.apache.kafka.common.serialization.ByteArrayDeserializer"

D. "org.apache.kafka.common.serialization.StringSerializer"

Solution

1. Step 1: Identify deserializer for String keys

   The correct class for String deserialization is StringDeserializer.

2. Step 2: Check option correctness

   "org.apache.kafka.common.serialization.StringDeserializer" uses StringDeserializer, which matches the key.deserializer property.

3. Final Answer:

   "org.apache.kafka.common.serialization.StringDeserializer" -> Option A

4. Quick Check:

   String deserializer = StringDeserializer [OK]

Hint: Deserializer ends with 'Deserializer', not 'Serializer' [OK]

Common Mistakes:

• Using Serializer instead of Deserializer
• Mixing IntegerSerializer with deserializer property
• Choosing ByteArrayDeserializer for String data

3.

Given the following Kafka consumer code snippet, what will be the output if the message value is a JSON string but the deserializer is set to StringDeserializer?

props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
ConsumerRecord<String, String> record = consumer.poll(Duration.ofMillis(100)).iterator().next();
System.out.println(record.value());

medium

A. Binary data printed as garbage characters

B. The JSON string as plain text

C. Null value printed

D. A deserialization error exception

Solution

1. Step 1: Understand StringDeserializer behavior

   StringDeserializer converts bytes to a UTF-8 string without parsing JSON.

2. Step 2: Output of printing record.value()

   The JSON string is printed as plain text exactly as received.

3. Final Answer:

   The JSON string as plain text -> Option B

4. Quick Check:

   StringDeserializer outputs raw string [OK]

Hint: StringDeserializer prints raw string, no JSON parsing [OK]

Common Mistakes:

• Expecting automatic JSON parsing
• Thinking deserialization fails on JSON strings
• Assuming binary garbage output

4.

What is the likely cause of this error in a Kafka consumer?

org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition

Given the consumer uses IntegerDeserializer but the producer sends String keys.

medium

A. Mismatch between producer's String keys and consumer's IntegerDeserializer

B. Kafka topic does not exist

C. Consumer group ID is missing

D. Network connection failure

Solution

1. Step 1: Analyze deserialization error cause

   The error occurs when deserializer cannot convert bytes to expected type.

2. Step 2: Check key type mismatch

   Producer sends String keys, but consumer expects Integer keys, causing failure.

3. Final Answer:

   Mismatch between producer's String keys and consumer's IntegerDeserializer -> Option A

4. Quick Check:

   Deserializer mismatch = SerializationException [OK]

Hint: Producer and consumer deserializers must match data type [OK]

Common Mistakes:

• Ignoring type mismatch between producer and consumer
• Blaming unrelated network or topic errors
• Confusing serialization with deserialization errors

5.

You want to consume messages where the value is a JSON object representing a user with fields id (int) and name (string). Which deserialization approach is best to convert the Kafka message value into a User Java object?

hard

A. Use ByteArrayDeserializer and manually convert bytes to User

B. Use StringDeserializer and cast the string to User

C. Use IntegerDeserializer for the value

D. Use a custom deserializer that parses JSON into a User object

Solution

1. Step 1: Understand data format and target type

   The message value is JSON, and we want a User Java object.

2. Step 2: Choose deserialization method

   A custom deserializer can parse JSON and create User instances directly.

3. Step 3: Evaluate other options

   StringDeserializer returns raw JSON string, casting won't convert to User. IntegerDeserializer is wrong type. ByteArrayDeserializer requires manual parsing.

4. Final Answer:

   Use a custom deserializer that parses JSON into a User object -> Option D

5. Quick Check:

   Custom deserializer = JSON to object [OK]

Hint: Custom deserializer parses JSON to Java object directly [OK]

Common Mistakes:

• Casting String to object without parsing
• Using wrong deserializer type
• Manually parsing bytes instead of using deserializer