Serialization (String, JSON, Avro) in Kafka - Mini Project: Build & Apply

Serialization with String, JSON, and Avro in Kafka

📖 Scenario: You are working on a messaging system where you need to send user information through Kafka. Different services expect data in different formats: plain string, JSON, and Avro.

🎯 Goal: Build a simple Kafka producer that serializes user data into three formats: String, JSON, and Avro, and sends them to different Kafka topics.

📋 What You'll Learn

Create a user data dictionary with exact fields and values

Define a Kafka topic name for each serialization format

Serialize the user data as a string, JSON, and Avro

Print the serialized data before sending

💡 Why This Matters

🌍 Real World

Kafka is widely used to send messages between services. Different services may require data in different formats like string, JSON, or Avro for compatibility and efficiency.

💼 Career

Understanding serialization formats and how to produce messages in Kafka is essential for backend developers, data engineers, and anyone working with real-time data pipelines.

1

Create user data dictionary

Create a dictionary called user_data with these exact entries: 'id': 101, 'name': 'Alice', 'email': '[e-mail]'

Kafka

# Create the user_data dictionary with id, name, and email
# Your code here

Hint

Use curly braces to create a dictionary with keys and values exactly as shown.

2

Define Kafka topic names

Create three string variables called string_topic, json_topic, and avro_topic with values 'user-string', 'user-json', and 'user-avro' respectively.

Kafka

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
# Define the Kafka topic names for string, JSON, and Avro
# Your code here

Hint

Assign the exact topic names as string values to the variables.

3

Serialize user data as string, JSON, and Avro

Import the json module. Create three variables: string_data by converting user_data to a string using str(), json_data by converting user_data to JSON string using json.dumps(), and avro_data by simulating Avro serialization as a byte string b'avro_serialized_data'.

Kafka

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
string_topic = 'user-string'
json_topic = 'user-json'
avro_topic = 'user-avro'

# Import json and serialize user_data to string_data, json_data, and avro_data
# Your code here

Hint

Use str() for string, json.dumps() for JSON, and simulate Avro with a byte string.

4

Print serialized data

Print the variables string_data, json_data, and avro_data each on a separate line.

Kafka

import json

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
string_topic = 'user-string'
json_topic = 'user-json'
avro_topic = 'user-avro'

string_data = str(user_data)
json_data = json.dumps(user_data)
avro_data = b'avro_serialized_data'

# Print string_data, json_data, and avro_data each on its own line
# Your code here

Hint

Use three print statements, one for each serialized variable.

Practice

1. What is the main purpose of serialization in Kafka?

easy

A. To convert data into a format suitable for transmission or storage

B. To encrypt data for security

C. To compress data to save space

D. To validate data against a schema

Solution

1. Step 1: Understand serialization concept

   Serialization means converting data into a format that can be sent or saved easily.

2. Step 2: Identify Kafka's use of serialization

   Kafka uses serialization to prepare data for sending between producers and consumers.

3. Final Answer:

   To convert data into a format suitable for transmission or storage -> Option A

4. Quick Check:

   Serialization = data format conversion [OK]

Hint: Serialization means data format conversion for sending [OK]

Common Mistakes:

• Confusing serialization with encryption
• Thinking serialization compresses data
• Assuming serialization validates data

2. Which of the following is the correct way to configure a Kafka producer to use JSON serialization in Java?

easy

A. props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer");

B. props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

C. props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

D. props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");

Solution

1. Step 1: Identify JSON serializer class

   The JSON serializer class in Kafka Connect is "org.apache.kafka.connect.json.JsonSerializer".

2. Step 2: Match configuration property

   The producer property for value serializer is "value.serializer" and should be set to the JSON serializer class.

3. Final Answer:

   props.put("value.serializer", "org.apache.kafka.connect.json.JsonSerializer"); -> Option A

4. Quick Check:

   JSON serializer = org.apache.kafka.connect.json.JsonSerializer [OK]

Hint: JSON serializer class is in kafka.connect.json package [OK]

Common Mistakes:

• Using StringSerializer for JSON data
• Confusing Avro serializer with JSON serializer
• Using ByteArraySerializer for JSON

3. Given the following Kafka producer code snippet using Avro serialization, what will be the output if the schema registry URL is incorrect?

props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://wrong-url:8081");
producer.send(new ProducerRecord<>("topic", avroRecord));

medium

A. NullPointerException during serialization

B. Message sent successfully without errors

C. SerializationException due to schema registry connection failure

D. Producer silently drops the message

Solution

1. Step 1: Understand Avro serialization dependency

   Avro serialization requires connecting to the schema registry to fetch schemas.

2. Step 2: Analyze effect of wrong schema registry URL

   If the URL is wrong, the serializer cannot fetch the schema and throws a SerializationException.

3. Final Answer:

   SerializationException due to schema registry connection failure -> Option C

4. Quick Check:

   Wrong schema URL = SerializationException [OK]

Hint: Avro needs schema registry; wrong URL causes SerializationException [OK]

Common Mistakes:

• Expecting message to send without error
• Thinking NullPointerException occurs
• Assuming message is dropped silently

4. Identify the error in this Kafka producer configuration for String serialization:

Properties props = new Properties();
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("bootstrap.servers", "localhost:9092");
Producer producer = new KafkaProducer<>(props);

medium

A. Incorrect generic types in KafkaProducer declaration

B. Using StringDeserializer instead of StringSerializer for value.serializer

C. Missing bootstrap.servers configuration

D. No error, configuration is correct

Solution

1. Step 1: Check serializer and deserializer usage

   Producer requires serializers for key and value, not deserializers.

2. Step 2: Identify wrong class usage

   Value serializer is incorrectly set to StringDeserializer instead of StringSerializer.

3. Final Answer:

   Using StringDeserializer instead of StringSerializer for value.serializer -> Option B

4. Quick Check:

   Producer needs serializers, not deserializers [OK]

Hint: Producer uses serializers, not deserializers [OK]

Common Mistakes:

• Mixing serializer and deserializer classes
• Forgetting bootstrap.servers (not asked here)
• Incorrect generic types (not the main error)

5. You want to send complex nested data with schema evolution support in Kafka. Which serialization method should you choose and why?

hard

A. ByteArray serialization, because it handles any data type

B. JSON serialization, because it supports nested data and is human-readable

C. String serialization, because it is simple and fast

D. Avro serialization, because it supports schema evolution and compact binary format

Solution

1. Step 1: Understand requirements for nested data and schema evolution

   Complex nested data needs a format that supports schemas and changes over time.

2. Step 2: Compare serialization options

   String is simple but no schema support; JSON supports nested but no schema evolution; Avro supports both schema evolution and compact binary format.

3. Final Answer:

   Avro serialization, because it supports schema evolution and compact binary format -> Option D

4. Quick Check:

   Schema evolution + nested data = Avro [OK]

Hint: Avro supports schema evolution and nested data best [OK]

Common Mistakes:

• Choosing String for complex data
• Assuming JSON supports schema evolution
• Using ByteArray without schema control