JSON Schema and Protobuf support in Kafka - Mini Project: Build & Apply

JSON Schema and Protobuf Support with Kafka

📖 Scenario: You are working on a messaging system using Kafka. You want to send and receive messages with structured data. To ensure data consistency, you will use JSON Schema and Protobuf formats for your messages.This project will guide you through creating data, configuring schema types, serializing messages, and printing the serialized output.

🎯 Goal: Build a simple Kafka message producer that creates data, configures the schema type (JSON Schema or Protobuf), serializes the data accordingly, and prints the serialized message.

📋 What You'll Learn

Create a data dictionary with specific fields

Set a variable to choose the schema type ('json' or 'protobuf')

Serialize the data based on the chosen schema type

Print the serialized message output

💡 Why This Matters

🌍 Real World

Kafka is widely used for messaging in distributed systems. Using JSON Schema and Protobuf helps keep data consistent and easy to parse across different services.

💼 Career

Understanding how to serialize data with JSON Schema and Protobuf is important for backend developers, data engineers, and anyone working with event-driven architectures.

1

Create the data dictionary

Create a dictionary called message_data with these exact entries: 'id': 101, 'name': 'Alice', 'active': True

Kafka

# Create the dictionary message_data with id, name, and active keys
# Your code here

Hint

Use curly braces to create a dictionary with keys and values exactly as shown.

2

Set the schema type variable

Create a variable called schema_type and set it to the string 'json'

Kafka

message_data = {'id': 101, 'name': 'Alice', 'active': True}
# Set schema_type to 'json'
# Your code here

Hint

Assign the string 'json' to the variable schema_type.

3

Serialize the data based on schema type

Write code to create a variable serialized_message that stores the JSON string of message_data if schema_type is 'json'. Otherwise, set serialized_message to the string 'protobuf_serialized'.

Kafka

message_data = {'id': 101, 'name': 'Alice', 'active': True}
schema_type = 'json'
# Serialize message_data to JSON string if schema_type is 'json', else set to 'protobuf_serialized'
# Your code here

Hint

Use json.dumps() to convert the dictionary to a JSON string when schema_type is 'json'. Use an if-else statement.

4

Print the serialized message

Write a print statement to display the value of serialized_message

Kafka

import json

message_data = {'id': 101, 'name': 'Alice', 'active': True}
schema_type = 'json'

if schema_type == 'json':
    serialized_message = json.dumps(message_data)
else:
    serialized_message = 'protobuf_serialized'

# Print the serialized_message
# Your code here

Hint

Use print(serialized_message) to show the serialized data.

Practice

1. What is the main purpose of using JSON Schema or Protobuf with Kafka messages?

easy

A. To define the structure and format of messages so producers and consumers agree

B. To encrypt Kafka messages for security

C. To speed up Kafka message delivery by skipping validation

D. To store Kafka messages permanently in a database

Solution

1. Step 1: Understand message format agreement

   JSON Schema and Protobuf define how messages look so both sender and receiver understand the data.

2. Step 2: Identify the main purpose

   This agreement prevents errors and confusion when reading messages.

3. Final Answer:

   To define the structure and format of messages so producers and consumers agree -> Option A

4. Quick Check:

   Message format agreement = C [OK]

Hint: They define message format so systems agree on data [OK]

Common Mistakes:

• Thinking JSON Schema encrypts messages
• Believing Protobuf speeds delivery by skipping validation
• Confusing schema with storage

2. Which of the following is the correct way to specify a Protobuf schema for a Kafka message?

easy

A. { 'name': 'string', 'id': 'int' }

B. schema { field name: string; field id: int; }

C. stringint

D. message Person { string name = 1; int32 id = 2; }

Solution

1. Step 1: Recognize Protobuf syntax

   Protobuf uses the 'message' keyword with fields and their types and numbers.

2. Step 2: Match the correct syntax

   message Person { string name = 1; int32 id = 2; } shows correct Protobuf message definition syntax.

3. Final Answer:

   message Person { string name = 1; int32 id = 2; } -> Option D

4. Quick Check:

   Protobuf syntax = B [OK]

Hint: Protobuf uses 'message' keyword and numbered fields [OK]

Common Mistakes:

• Confusing JSON or XML syntax with Protobuf
• Missing field numbers in Protobuf
• Using invalid keywords like 'schema'

3. Given this JSON Schema snippet for a Kafka message:

{
  "type": "object",
  "properties": {
    "userId": {"type": "string"},
    "age": {"type": "integer"}
  },
  "required": ["userId"]
}

Which message is valid according to this schema?

medium

A. {"userId": "abc123", "age": 30}

B. {"age": 30}

C. {"userId": 123, "age": 30}

D. {"userId": "abc123", "age": "30"}

Solution

1. Step 1: Check required fields and types

   The schema requires 'userId' as a string and 'age' as an integer (optional).

2. Step 2: Validate each option

   {"userId": "abc123", "age": 30} has 'userId' as string and 'age' as integer, so it is valid.

3. Final Answer:

   {"userId": "abc123", "age": 30} -> Option A

4. Quick Check:

   Required string userId and integer age = A [OK]

Hint: Required fields must be present with correct types [OK]

Common Mistakes:

• Missing required 'userId' field
• Using wrong data types for fields
• Confusing string and integer types

4. You have this Protobuf schema:

message Event {
  string id = 1;
  int32 count = 2;
}

Which of the following is CORRECT about using it with Kafka?

medium

A. Using 'int32' instead of 'integer' causes error

B. Field numbers must be unique and start from 1

C. Missing a root 'schema' keyword causes failure

D. Field names cannot be strings

Solution

1. Step 1: Check Protobuf field numbering rules

   Field numbers must be unique and start from 1, which is true here.

2. Step 2: Identify the correct statement

   Field numbers must be unique and start from 1 correctly describes the field numbering rule. Options B, C, D are incorrect statements.

3. Final Answer:

   Field numbers must be unique and start from 1 -> Option B

4. Quick Check:

   Protobuf field numbering rule = A [OK]

Hint: Field numbers must be unique and start at 1 [OK]

Common Mistakes:

• Thinking 'int32' is invalid
• Expecting a 'schema' keyword in Protobuf
• Confusing field names with types

5. You want to optimize Kafka message size and speed. Which schema choice is best and why?

hard

A. Use JSON Schema because it is human-readable and easy to debug

B. Use XML schema because it supports complex data types

C. Use Protobuf because it is compact and faster to serialize/deserialize

D. Use plain text messages without schema for maximum speed

Solution

1. Step 1: Compare JSON Schema and Protobuf

   JSON Schema is readable but larger; Protobuf is compact and faster.

2. Step 2: Choose based on optimization goal

   For smaller size and speed, Protobuf is better.

3. Final Answer:

   Use Protobuf because it is compact and faster to serialize/deserialize -> Option C

4. Quick Check:

   Compact and fast = D [OK]

Hint: Protobuf is compact and faster than JSON Schema [OK]

Common Mistakes:

• Choosing JSON Schema for speed optimization
• Ignoring schema benefits and using plain text
• Confusing XML with Protobuf advantages