JSON Schema and Protobuf support in Kafka - Step-by-Step Execution

Process Flow - JSON Schema and Protobuf support

Start: Define Schema

↓

Choose Format: JSON Schema or Protobuf

↓

Register Schema in Registry

↓

Produce Message with Schema

↓

Kafka Broker stores message

↓

Consumer fetches message

↓

Deserialize using Schema

↓

Process Data

↓

End

Shows the flow of using JSON Schema or Protobuf with Kafka: define schema, register it, produce and consume messages with schema validation.

Execution Sample

Kafka

schema = load_schema('user.proto')
producer.send(topic, value=user_data, schema=schema)
consumer.poll()
user = consumer.deserialize(schema)

This code loads a Protobuf schema, sends a message with that schema, then consumes and deserializes it.

Process Table

Step	Action	Input/Condition	Result/Output
1	Load schema	'user.proto'	Schema object created
2	Register schema	Schema object	Schema registered in registry
3	Produce message	user_data with schema	Message sent to Kafka topic
4	Kafka stores message	Message with schema	Message stored with schema ID
5	Consumer polls	Topic and schema ID	Message fetched from Kafka
6	Deserialize message	Fetched message and schema	User object created
7	Process data	User object	Data ready for application
8	End	No further input	Process complete

💡 Process ends after message is deserialized and data is ready

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 6	Final
schema	null	Schema object	Schema object	Schema object	Schema object
user_data	Provided	Provided	Sent with schema	N/A	N/A
message	N/A	N/A	Sent to Kafka	Fetched from Kafka	Deserialized to user object
user	N/A	N/A	N/A	Created from message	Used in processing

Key Moments - 3 Insights

Why do we need to register the schema before sending messages?

Registering the schema assigns it an ID so Kafka can validate and deserialize messages correctly, as shown in step 2 of the execution_table.

What happens if the consumer uses a different schema than the producer?

Deserialization will fail or produce incorrect data because the schema IDs won't match, as seen in step 6 where the schema is needed to create the user object.

Can we use both JSON Schema and Protobuf with Kafka interchangeably?

Yes, but each message must follow the registered schema format; the process flow is the same but schemas differ in format (step 1 and 2).

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 3?

A User object created

B Schema registered in registry

C Message sent to Kafka topic

D Message fetched from Kafka

💡 Hint

Check the 'Result/Output' column for step 3 in the execution_table.

At which step does the consumer deserialize the message?

A Step 6

B Step 5

C Step 4

D Step 7

💡 Hint

Look for 'Deserialize message' action in the execution_table.

If the schema is not registered, what will likely happen during message production?

A Message will be sent without schema validation

B Message production will fail

C Consumer will deserialize message successfully

D Kafka will auto-generate a schema

💡 Hint

Refer to step 2 and 3 in the execution_table about schema registration importance.

Concept Snapshot

Kafka supports JSON Schema and Protobuf for message validation.
Define and register schema before producing messages.
Messages include schema ID for validation.
Consumers use schema to deserialize messages.
Ensures data consistency and compatibility.

Full Transcript

This visual execution shows how Kafka uses JSON Schema or Protobuf to ensure messages are structured and validated. First, a schema is defined and registered in a schema registry. Then, when producing messages, the schema is attached so Kafka can store the message with a schema ID. Consumers fetch messages and use the schema to deserialize them back into usable objects. This process helps keep data consistent and prevents errors from mismatched formats.