JSON Schema and Protobuf support in Kafka - Mini Project: Build & Apply

JSON Schema and Protobuf Support with Kafka

📖 Scenario: You are working on a messaging system using Kafka. You want to send and receive messages with structured data. To ensure data consistency, you will use JSON Schema and Protobuf formats for your messages.This project will guide you through creating data, configuring schema types, serializing messages, and printing the serialized output.

🎯 Goal: Build a simple Kafka message producer that creates data, configures the schema type (JSON Schema or Protobuf), serializes the data accordingly, and prints the serialized message.

📋 What You'll Learn

Create a data dictionary with specific fields

Set a variable to choose the schema type ('json' or 'protobuf')

Serialize the data based on the chosen schema type

Print the serialized message output

💡 Why This Matters

🌍 Real World

Kafka is widely used for messaging in distributed systems. Using JSON Schema and Protobuf helps keep data consistent and easy to parse across different services.

💼 Career

Understanding how to serialize data with JSON Schema and Protobuf is important for backend developers, data engineers, and anyone working with event-driven architectures.

1

Create the data dictionary

Create a dictionary called message_data with these exact entries: 'id': 101, 'name': 'Alice', 'active': True

Kafka

# Create the dictionary message_data with id, name, and active keys
# Your code here

Need a hint?

Use curly braces to create a dictionary with keys and values exactly as shown.

2

Set the schema type variable

Create a variable called schema_type and set it to the string 'json'

Kafka

message_data = {'id': 101, 'name': 'Alice', 'active': True}
# Set schema_type to 'json'
# Your code here

Need a hint?

Assign the string 'json' to the variable schema_type.

3

Serialize the data based on schema type

Write code to create a variable serialized_message that stores the JSON string of message_data if schema_type is 'json'. Otherwise, set serialized_message to the string 'protobuf_serialized'.

Kafka

message_data = {'id': 101, 'name': 'Alice', 'active': True}
schema_type = 'json'
# Serialize message_data to JSON string if schema_type is 'json', else set to 'protobuf_serialized'
# Your code here

Need a hint?

Use json.dumps() to convert the dictionary to a JSON string when schema_type is 'json'. Use an if-else statement.

4

Print the serialized message

Write a print statement to display the value of serialized_message

Kafka

import json

message_data = {'id': 101, 'name': 'Alice', 'active': True}
schema_type = 'json'

if schema_type == 'json':
    serialized_message = json.dumps(message_data)
else:
    serialized_message = 'protobuf_serialized'

# Print the serialized_message
# Your code here

Need a hint?

Use print(serialized_message) to show the serialized data.