Schema Registry concept in Kafka - Step-by-Step Execution

Process Flow - Schema Registry concept

Producer sends data

↓

Schema Registry checks schema

↓

Schema valid

↓

Data sent

↓

Consumer fetches schema

↓

Consumer deserializes data

Data flows from producer to consumer with schema validation in between to ensure data format consistency.

Execution Sample

Kafka

1. Producer sends data with schema ID
2. Schema Registry validates schema
3. If valid, data is stored
4. Consumer fetches schema by ID
5. Consumer deserializes data using schema

Shows how data is validated and serialized/deserialized using Schema Registry in Kafka.

Process Table

Step	Action	Input	Schema Registry Response	Result
1	Producer sends data with schema ID	Data + Schema ID 1	Checks schema ID 1	Schema found, validation passed
2	Schema Registry validates schema	Schema ID 1	Schema valid	Data accepted and stored
3	Consumer requests schema by ID	Schema ID 1	Returns schema definition	Consumer gets schema
4	Consumer deserializes data	Data + Schema	Uses schema to parse data	Data correctly deserialized
5	Producer sends data with unknown schema	Data + Schema ID 99	Schema not found	Error returned to producer

💡 Execution stops when schema validation fails or data is successfully processed.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5
Schema ID	None	1	1	1	1	99
Data	None	Sent with schema 1	Validated	Stored	Deserialized	Sent with unknown schema
Schema Registry Response	None	Checking	Valid	Returned schema	Used for deserialization	Error

Key Moments - 3 Insights

Why does the producer get an error when sending data with an unknown schema ID?

Because the Schema Registry cannot find the schema ID (see execution_table step 5), it rejects the data to prevent format mismatch.

How does the consumer know how to read the data?

The consumer fetches the schema by ID from the Schema Registry (execution_table step 3) and uses it to deserialize the data correctly.

What happens if the schema changes after data is produced?

The Schema Registry manages schema versions; consumers use the schema ID embedded in data to deserialize with the correct version, ensuring compatibility.

Visual Quiz - 3 Questions

Look at the execution table, what is the Schema Registry response at step 2?

A Schema not found

B Schema valid

C Error

D Schema updated

💡 Hint

Check the 'Schema Registry Response' column at step 2 in the execution_table.

At which step does the consumer get the schema definition?

A Step 3

B Step 1

C Step 5

D Step 4

💡 Hint

Look at the 'Action' column for when the consumer requests the schema.

If the producer sends data with a valid schema ID, what will happen according to the table?

A Error returned

B Data rejected

C Data accepted and stored

D Schema updated

💡 Hint

Refer to the 'Result' column at step 2 in the execution_table.

Concept Snapshot

Schema Registry ensures data format consistency in Kafka.
Producer sends data with schema ID.
Registry validates schema before accepting data.
Consumer fetches schema by ID to deserialize.
Prevents errors from schema mismatches.

Full Transcript

In Kafka, the Schema Registry acts as a gatekeeper for data formats. When a producer sends data, it includes a schema ID. The Schema Registry checks if this schema is valid. If valid, the data is accepted and stored. If not, the producer gets an error. Consumers then request the schema by ID from the registry to correctly read and deserialize the data. This process ensures that producers and consumers agree on data structure, preventing errors and data corruption.

Practice

1. What is the main purpose of a Schema Registry in Kafka?

easy

A. To encrypt Kafka messages during transmission

B. To monitor Kafka broker performance

C. To route messages between Kafka topics

D. To store and manage data schemas for Kafka messages

Solution

1. Step 1: Understand the role of Schema Registry

   Schema Registry stores and manages the structure (schema) of data used in Kafka messages.

2. Step 2: Compare with other Kafka components

   Monitoring, routing, and encryption are handled by other Kafka tools, not Schema Registry.

3. Final Answer:

   To store and manage data schemas for Kafka messages -> Option D

4. Quick Check:

   Schema Registry = data schema storage [OK]

Hint: Schema Registry manages data formats, not Kafka internals [OK]

Common Mistakes:

• Confusing Schema Registry with Kafka brokers
• Thinking it handles message routing
• Assuming it encrypts messages

2. Which of the following is the correct way to register a schema using Kafka Schema Registry API?

easy

A. DELETE /topics/{topic}/schemas with schema in YAML

B. GET /schemas/{id} with schema in XML

C. POST /subjects/{subject}/versions with schema in JSON

D. PUT /brokers/{brokerId}/schemas with schema in CSV

Solution

1. Step 1: Identify the correct HTTP method and endpoint

   Registering a schema uses POST on /subjects/{subject}/versions with JSON schema.

2. Step 2: Check other options for correctness

   GET retrieves schemas, DELETE removes topics (not schemas), PUT on brokers is invalid for schema registration.

3. Final Answer:

   POST /subjects/{subject}/versions with schema in JSON -> Option C

4. Quick Check:

   Register schema = POST with JSON [OK]

Hint: Register schemas with POST and JSON format [OK]

Common Mistakes:

• Using GET instead of POST to register
• Confusing schema formats (JSON vs XML/YAML/CSV)
• Using wrong API endpoints

3. Given this code snippet using Kafka Schema Registry client, what will be the output?

schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
print(schema_id)

medium

A. An integer ID representing the registered schema

B. The original schema string printed

C. A JSON object with schema details

D. A runtime error due to missing client method

Solution

1. Step 1: Understand the register_schema method

   This method registers the schema and returns an integer ID assigned by Schema Registry.

2. Step 2: Analyze the print statement

   Printing schema_id outputs the integer ID, not the schema string or JSON object.

3. Final Answer:

   An integer ID representing the registered schema -> Option A

4. Quick Check:

   Register returns schema ID integer [OK]

Hint: Register returns schema ID number, not schema text [OK]

Common Mistakes:

• Expecting schema string instead of ID
• Confusing output with JSON object
• Assuming method does not exist

4. Identify the error in this Kafka Schema Registry usage snippet:

schema = '{"type":"record","name":"User","fields":[{"name":"age","type":"int"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
client.register_schema(subject, schema)

medium

A. Registering the same schema twice causes an error

B. No error; registering same schema returns same ID

C. Schema string is not valid JSON

D. Missing schema version in registration call

Solution

1. Step 1: Check schema validity

   The schema string is valid JSON describing a record with an int field.

2. Step 2: Understand Schema Registry behavior on duplicate schemas

   Registering the same schema again returns the same schema ID without error.

3. Final Answer:

   No error; registering same schema returns same ID -> Option B

4. Quick Check:

   Duplicate schema registration returns same ID [OK]

Hint: Duplicate schema registration is safe and returns same ID [OK]

Common Mistakes:

• Assuming duplicate registration causes error
• Thinking schema string is invalid JSON
• Expecting version parameter in register call

5. You want to evolve a Kafka message schema by adding a new optional field without breaking existing consumers. Which Schema Registry feature helps achieve this safely?

hard

A. Schema compatibility settings (e.g., BACKWARD compatibility)

B. Schema encryption for secure transmission

C. Schema version rollback to previous schema

D. Schema validation disabling for faster registration

Solution

1. Step 1: Understand schema evolution needs

   Adding a new optional field requires compatibility so old consumers still work.

2. Step 2: Identify Schema Registry feature for compatibility

   Schema compatibility settings like BACKWARD ensure new schemas don't break old consumers.

3. Step 3: Eliminate other options

   Encryption, rollback, or disabling validation do not help safe schema evolution.

4. Final Answer:

   Schema compatibility settings (e.g., BACKWARD compatibility) -> Option A

5. Quick Check:

   Compatibility settings enable safe schema evolution [OK]

Hint: Use compatibility settings to evolve schemas safely [OK]

Common Mistakes:

• Confusing encryption with compatibility
• Thinking rollback helps forward compatibility
• Disabling validation breaks schema safety