Schema Registry concept in Kafka - Step-by-Step Execution

Process Flow - Schema Registry concept

Producer sends data

↓

Schema Registry checks schema

↓

Schema valid

↓

Data sent

↓

Consumer fetches schema

↓

Consumer deserializes data

Data flows from producer to consumer with schema validation in between to ensure data format consistency.

Execution Sample

Kafka

1. Producer sends data with schema ID
2. Schema Registry validates schema
3. If valid, data is stored
4. Consumer fetches schema by ID
5. Consumer deserializes data using schema

Shows how data is validated and serialized/deserialized using Schema Registry in Kafka.

Process Table

Step	Action	Input	Schema Registry Response	Result
1	Producer sends data with schema ID	Data + Schema ID 1	Checks schema ID 1	Schema found, validation passed
2	Schema Registry validates schema	Schema ID 1	Schema valid	Data accepted and stored
3	Consumer requests schema by ID	Schema ID 1	Returns schema definition	Consumer gets schema
4	Consumer deserializes data	Data + Schema	Uses schema to parse data	Data correctly deserialized
5	Producer sends data with unknown schema	Data + Schema ID 99	Schema not found	Error returned to producer

💡 Execution stops when schema validation fails or data is successfully processed.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5
Schema ID	None	1	1	1	1	99
Data	None	Sent with schema 1	Validated	Stored	Deserialized	Sent with unknown schema
Schema Registry Response	None	Checking	Valid	Returned schema	Used for deserialization	Error

Key Moments - 3 Insights

Why does the producer get an error when sending data with an unknown schema ID?

Because the Schema Registry cannot find the schema ID (see execution_table step 5), it rejects the data to prevent format mismatch.

How does the consumer know how to read the data?

The consumer fetches the schema by ID from the Schema Registry (execution_table step 3) and uses it to deserialize the data correctly.

What happens if the schema changes after data is produced?

The Schema Registry manages schema versions; consumers use the schema ID embedded in data to deserialize with the correct version, ensuring compatibility.

Visual Quiz - 3 Questions

Look at the execution table, what is the Schema Registry response at step 2?

A Schema not found

B Schema valid

C Error

D Schema updated

💡 Hint

Check the 'Schema Registry Response' column at step 2 in the execution_table.

At which step does the consumer get the schema definition?

A Step 3

B Step 1

C Step 5

D Step 4

💡 Hint

Look at the 'Action' column for when the consumer requests the schema.

If the producer sends data with a valid schema ID, what will happen according to the table?

A Error returned

B Data rejected

C Data accepted and stored

D Schema updated

💡 Hint

Refer to the 'Result' column at step 2 in the execution_table.

Concept Snapshot

Schema Registry ensures data format consistency in Kafka.
Producer sends data with schema ID.
Registry validates schema before accepting data.
Consumer fetches schema by ID to deserialize.
Prevents errors from schema mismatches.

Full Transcript

In Kafka, the Schema Registry acts as a gatekeeper for data formats. When a producer sends data, it includes a schema ID. The Schema Registry checks if this schema is valid. If valid, the data is accepted and stored. If not, the producer gets an error. Consumers then request the schema by ID from the registry to correctly read and deserialize the data. This process ensures that producers and consumers agree on data structure, preventing errors and data corruption.