Serialization (String, JSON, Avro) in Kafka - Mini Project: Build & Apply

Serialization with String, JSON, and Avro in Kafka

📖 Scenario: You are working on a messaging system where you need to send user information through Kafka. Different services expect data in different formats: plain string, JSON, and Avro.

🎯 Goal: Build a simple Kafka producer that serializes user data into three formats: String, JSON, and Avro, and sends them to different Kafka topics.

📋 What You'll Learn

Create a user data dictionary with exact fields and values

Define a Kafka topic name for each serialization format

Serialize the user data as a string, JSON, and Avro

Print the serialized data before sending

💡 Why This Matters

🌍 Real World

Kafka is widely used to send messages between services. Different services may require data in different formats like string, JSON, or Avro for compatibility and efficiency.

💼 Career

Understanding serialization formats and how to produce messages in Kafka is essential for backend developers, data engineers, and anyone working with real-time data pipelines.

1

Create user data dictionary

Create a dictionary called user_data with these exact entries: 'id': 101, 'name': 'Alice', 'email': '[e-mail]'

Kafka

# Create the user_data dictionary with id, name, and email
# Your code here

Need a hint?

Use curly braces to create a dictionary with keys and values exactly as shown.

2

Define Kafka topic names

Create three string variables called string_topic, json_topic, and avro_topic with values 'user-string', 'user-json', and 'user-avro' respectively.

Kafka

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
# Define the Kafka topic names for string, JSON, and Avro
# Your code here

Need a hint?

Assign the exact topic names as string values to the variables.

3

Serialize user data as string, JSON, and Avro

Import the json module. Create three variables: string_data by converting user_data to a string using str(), json_data by converting user_data to JSON string using json.dumps(), and avro_data by simulating Avro serialization as a byte string b'avro_serialized_data'.

Kafka

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
string_topic = 'user-string'
json_topic = 'user-json'
avro_topic = 'user-avro'

# Import json and serialize user_data to string_data, json_data, and avro_data
# Your code here

Need a hint?

Use str() for string, json.dumps() for JSON, and simulate Avro with a byte string.

4

Print serialized data

Print the variables string_data, json_data, and avro_data each on a separate line.

Kafka

import json

user_data = {'id': 101, 'name': 'Alice', 'email': '[e-mail]'}
string_topic = 'user-string'
json_topic = 'user-json'
avro_topic = 'user-avro'

string_data = str(user_data)
json_data = json.dumps(user_data)
avro_data = b'avro_serialized_data'

# Print string_data, json_data, and avro_data each on its own line
# Your code here

Need a hint?

Use three print statements, one for each serialized variable.