JSON Schema and Protobuf support in Kafka - Step-by-Step Execution

Process Flow - JSON Schema and Protobuf support

Start: Define Schema

↓

Choose Format: JSON Schema or Protobuf

↓

Register Schema in Registry

↓

Produce Message with Schema

↓

Kafka Broker stores message

↓

Consumer fetches message

↓

Deserialize using Schema

↓

Process Data

↓

End

Shows the flow of using JSON Schema or Protobuf with Kafka: define schema, register it, produce and consume messages with schema validation.

Execution Sample

Kafka

schema = load_schema('user.proto')
producer.send(topic, value=user_data, schema=schema)
consumer.poll()
user = consumer.deserialize(schema)

This code loads a Protobuf schema, sends a message with that schema, then consumes and deserializes it.

Process Table

Step	Action	Input/Condition	Result/Output
1	Load schema	'user.proto'	Schema object created
2	Register schema	Schema object	Schema registered in registry
3	Produce message	user_data with schema	Message sent to Kafka topic
4	Kafka stores message	Message with schema	Message stored with schema ID
5	Consumer polls	Topic and schema ID	Message fetched from Kafka
6	Deserialize message	Fetched message and schema	User object created
7	Process data	User object	Data ready for application
8	End	No further input	Process complete

💡 Process ends after message is deserialized and data is ready

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 6	Final
schema	null	Schema object	Schema object	Schema object	Schema object
user_data	Provided	Provided	Sent with schema	N/A	N/A
message	N/A	N/A	Sent to Kafka	Fetched from Kafka	Deserialized to user object
user	N/A	N/A	N/A	Created from message	Used in processing

Key Moments - 3 Insights

Why do we need to register the schema before sending messages?

Registering the schema assigns it an ID so Kafka can validate and deserialize messages correctly, as shown in step 2 of the execution_table.

What happens if the consumer uses a different schema than the producer?

Deserialization will fail or produce incorrect data because the schema IDs won't match, as seen in step 6 where the schema is needed to create the user object.

Can we use both JSON Schema and Protobuf with Kafka interchangeably?

Yes, but each message must follow the registered schema format; the process flow is the same but schemas differ in format (step 1 and 2).

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 3?

A User object created

B Schema registered in registry

C Message sent to Kafka topic

D Message fetched from Kafka

💡 Hint

Check the 'Result/Output' column for step 3 in the execution_table.

At which step does the consumer deserialize the message?

A Step 6

B Step 5

C Step 4

D Step 7

💡 Hint

Look for 'Deserialize message' action in the execution_table.

If the schema is not registered, what will likely happen during message production?

A Message will be sent without schema validation

B Message production will fail

C Consumer will deserialize message successfully

D Kafka will auto-generate a schema

💡 Hint

Refer to step 2 and 3 in the execution_table about schema registration importance.

Concept Snapshot

Kafka supports JSON Schema and Protobuf for message validation.
Define and register schema before producing messages.
Messages include schema ID for validation.
Consumers use schema to deserialize messages.
Ensures data consistency and compatibility.

Full Transcript

This visual execution shows how Kafka uses JSON Schema or Protobuf to ensure messages are structured and validated. First, a schema is defined and registered in a schema registry. Then, when producing messages, the schema is attached so Kafka can store the message with a schema ID. Consumers fetch messages and use the schema to deserialize them back into usable objects. This process helps keep data consistent and prevents errors from mismatched formats.

Practice

1. What is the main purpose of using JSON Schema or Protobuf with Kafka messages?

easy

A. To define the structure and format of messages so producers and consumers agree

B. To encrypt Kafka messages for security

C. To speed up Kafka message delivery by skipping validation

D. To store Kafka messages permanently in a database

Solution

1. Step 1: Understand message format agreement

   JSON Schema and Protobuf define how messages look so both sender and receiver understand the data.

2. Step 2: Identify the main purpose

   This agreement prevents errors and confusion when reading messages.

3. Final Answer:

   To define the structure and format of messages so producers and consumers agree -> Option A

4. Quick Check:

   Message format agreement = C [OK]

Hint: They define message format so systems agree on data [OK]

Common Mistakes:

• Thinking JSON Schema encrypts messages
• Believing Protobuf speeds delivery by skipping validation
• Confusing schema with storage

2. Which of the following is the correct way to specify a Protobuf schema for a Kafka message?

easy

A. { 'name': 'string', 'id': 'int' }

B. schema { field name: string; field id: int; }

C. stringint

D. message Person { string name = 1; int32 id = 2; }

Solution

1. Step 1: Recognize Protobuf syntax

   Protobuf uses the 'message' keyword with fields and their types and numbers.

2. Step 2: Match the correct syntax

   message Person { string name = 1; int32 id = 2; } shows correct Protobuf message definition syntax.

3. Final Answer:

   message Person { string name = 1; int32 id = 2; } -> Option D

4. Quick Check:

   Protobuf syntax = B [OK]

Hint: Protobuf uses 'message' keyword and numbered fields [OK]

Common Mistakes:

• Confusing JSON or XML syntax with Protobuf
• Missing field numbers in Protobuf
• Using invalid keywords like 'schema'

3. Given this JSON Schema snippet for a Kafka message:

{
  "type": "object",
  "properties": {
    "userId": {"type": "string"},
    "age": {"type": "integer"}
  },
  "required": ["userId"]
}

Which message is valid according to this schema?

medium

A. {"userId": "abc123", "age": 30}

B. {"age": 30}

C. {"userId": 123, "age": 30}

D. {"userId": "abc123", "age": "30"}

Solution

1. Step 1: Check required fields and types

   The schema requires 'userId' as a string and 'age' as an integer (optional).

2. Step 2: Validate each option

   {"userId": "abc123", "age": 30} has 'userId' as string and 'age' as integer, so it is valid.

3. Final Answer:

   {"userId": "abc123", "age": 30} -> Option A

4. Quick Check:

   Required string userId and integer age = A [OK]

Hint: Required fields must be present with correct types [OK]

Common Mistakes:

• Missing required 'userId' field
• Using wrong data types for fields
• Confusing string and integer types

4. You have this Protobuf schema:

message Event {
  string id = 1;
  int32 count = 2;
}

Which of the following is CORRECT about using it with Kafka?

medium

A. Using 'int32' instead of 'integer' causes error

B. Field numbers must be unique and start from 1

C. Missing a root 'schema' keyword causes failure

D. Field names cannot be strings

Solution

1. Step 1: Check Protobuf field numbering rules

   Field numbers must be unique and start from 1, which is true here.

2. Step 2: Identify the correct statement

   Field numbers must be unique and start from 1 correctly describes the field numbering rule. Options B, C, D are incorrect statements.

3. Final Answer:

   Field numbers must be unique and start from 1 -> Option B

4. Quick Check:

   Protobuf field numbering rule = A [OK]

Hint: Field numbers must be unique and start at 1 [OK]

Common Mistakes:

• Thinking 'int32' is invalid
• Expecting a 'schema' keyword in Protobuf
• Confusing field names with types

5. You want to optimize Kafka message size and speed. Which schema choice is best and why?

hard

A. Use JSON Schema because it is human-readable and easy to debug

B. Use XML schema because it supports complex data types

C. Use Protobuf because it is compact and faster to serialize/deserialize

D. Use plain text messages without schema for maximum speed

Solution

1. Step 1: Compare JSON Schema and Protobuf

   JSON Schema is readable but larger; Protobuf is compact and faster.

2. Step 2: Choose based on optimization goal

   For smaller size and speed, Protobuf is better.

3. Final Answer:

   Use Protobuf because it is compact and faster to serialize/deserialize -> Option C

4. Quick Check:

   Compact and fast = D [OK]

Hint: Protobuf is compact and faster than JSON Schema [OK]

Common Mistakes:

• Choosing JSON Schema for speed optimization
• Ignoring schema benefits and using plain text
• Confusing XML with Protobuf advantages