Schema Registry concept in Kafka - Mini Project: Build & Apply

Schema Registry Concept with Kafka

📖 Scenario: You are working with Kafka to send messages between services. To keep data consistent, you use a Schema Registry that stores the structure of your messages. This helps producers and consumers agree on the data format.

🎯 Goal: Learn how to register a schema, configure a Kafka producer to use the Schema Registry, and produce a message that follows the schema.

📋 What You'll Learn

Create a schema definition in JSON format

Set a variable for the Schema Registry URL

Configure a Kafka producer to use the Schema Registry

Produce a message that matches the schema

Print the confirmation of message sent

💡 Why This Matters

🌍 Real World

Schema Registry ensures that all services agree on the data format, preventing errors and data corruption in distributed systems.

💼 Career

Understanding Schema Registry is essential for roles involving Kafka, data pipelines, and event-driven architectures.

1

Create the JSON schema definition

Create a variable called user_schema and assign it this exact JSON string representing a user schema: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}

Kafka

# Create the user_schema variable with the exact JSON string
# Your code here

Hint

Use single quotes outside and double quotes inside the JSON string.

2

Set the Schema Registry URL

Create a variable called schema_registry_url and set it to the string "http://localhost:8081"

Kafka

user_schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}'
# Create the schema_registry_url variable with the exact string "http://localhost:8081"
# Your code here

Hint

Use double quotes for the URL string.

3

Configure Kafka producer with Schema Registry

Create a dictionary called producer_config with keys "bootstrap.servers" set to "localhost:9092" and "schema.registry.url" set to the variable schema_registry_url

Kafka

user_schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}'
schema_registry_url = "http://localhost:8081"
# Create producer_config dictionary with keys "bootstrap.servers" and "schema.registry.url"
# Your code here

Hint

Use curly braces to create the dictionary and include the exact keys and values.

4

Produce a message and print confirmation

Write a print statement that outputs exactly "Message sent with schema User"

Kafka

user_schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}'
schema_registry_url = "http://localhost:8081"
producer_config = {
    "bootstrap.servers": "localhost:9092",
    "schema.registry.url": schema_registry_url
}
# Print the confirmation message exactly as "Message sent with schema User"
# Your code here

Hint

Use the print function with the exact string inside double quotes.

Practice

1. What is the main purpose of a Schema Registry in Kafka?

easy

A. To encrypt Kafka messages during transmission

B. To monitor Kafka broker performance

C. To route messages between Kafka topics

D. To store and manage data schemas for Kafka messages

Solution

1. Step 1: Understand the role of Schema Registry

   Schema Registry stores and manages the structure (schema) of data used in Kafka messages.

2. Step 2: Compare with other Kafka components

   Monitoring, routing, and encryption are handled by other Kafka tools, not Schema Registry.

3. Final Answer:

   To store and manage data schemas for Kafka messages -> Option D

4. Quick Check:

   Schema Registry = data schema storage [OK]

Hint: Schema Registry manages data formats, not Kafka internals [OK]

Common Mistakes:

• Confusing Schema Registry with Kafka brokers
• Thinking it handles message routing
• Assuming it encrypts messages

2. Which of the following is the correct way to register a schema using Kafka Schema Registry API?

easy

A. DELETE /topics/{topic}/schemas with schema in YAML

B. GET /schemas/{id} with schema in XML

C. POST /subjects/{subject}/versions with schema in JSON

D. PUT /brokers/{brokerId}/schemas with schema in CSV

Solution

1. Step 1: Identify the correct HTTP method and endpoint

   Registering a schema uses POST on /subjects/{subject}/versions with JSON schema.

2. Step 2: Check other options for correctness

   GET retrieves schemas, DELETE removes topics (not schemas), PUT on brokers is invalid for schema registration.

3. Final Answer:

   POST /subjects/{subject}/versions with schema in JSON -> Option C

4. Quick Check:

   Register schema = POST with JSON [OK]

Hint: Register schemas with POST and JSON format [OK]

Common Mistakes:

• Using GET instead of POST to register
• Confusing schema formats (JSON vs XML/YAML/CSV)
• Using wrong API endpoints

3. Given this code snippet using Kafka Schema Registry client, what will be the output?

schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
print(schema_id)

medium

A. An integer ID representing the registered schema

B. The original schema string printed

C. A JSON object with schema details

D. A runtime error due to missing client method

Solution

1. Step 1: Understand the register_schema method

   This method registers the schema and returns an integer ID assigned by Schema Registry.

2. Step 2: Analyze the print statement

   Printing schema_id outputs the integer ID, not the schema string or JSON object.

3. Final Answer:

   An integer ID representing the registered schema -> Option A

4. Quick Check:

   Register returns schema ID integer [OK]

Hint: Register returns schema ID number, not schema text [OK]

Common Mistakes:

• Expecting schema string instead of ID
• Confusing output with JSON object
• Assuming method does not exist

4. Identify the error in this Kafka Schema Registry usage snippet:

schema = '{"type":"record","name":"User","fields":[{"name":"age","type":"int"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
client.register_schema(subject, schema)

medium

A. Registering the same schema twice causes an error

B. No error; registering same schema returns same ID

C. Schema string is not valid JSON

D. Missing schema version in registration call

Solution

1. Step 1: Check schema validity

   The schema string is valid JSON describing a record with an int field.

2. Step 2: Understand Schema Registry behavior on duplicate schemas

   Registering the same schema again returns the same schema ID without error.

3. Final Answer:

   No error; registering same schema returns same ID -> Option B

4. Quick Check:

   Duplicate schema registration returns same ID [OK]

Hint: Duplicate schema registration is safe and returns same ID [OK]

Common Mistakes:

• Assuming duplicate registration causes error
• Thinking schema string is invalid JSON
• Expecting version parameter in register call

5. You want to evolve a Kafka message schema by adding a new optional field without breaking existing consumers. Which Schema Registry feature helps achieve this safely?

hard

A. Schema compatibility settings (e.g., BACKWARD compatibility)

B. Schema encryption for secure transmission

C. Schema version rollback to previous schema

D. Schema validation disabling for faster registration

Solution

1. Step 1: Understand schema evolution needs

   Adding a new optional field requires compatibility so old consumers still work.

2. Step 2: Identify Schema Registry feature for compatibility

   Schema compatibility settings like BACKWARD ensure new schemas don't break old consumers.

3. Step 3: Eliminate other options

   Encryption, rollback, or disabling validation do not help safe schema evolution.

4. Final Answer:

   Schema compatibility settings (e.g., BACKWARD compatibility) -> Option A

5. Quick Check:

   Compatibility settings enable safe schema evolution [OK]

Hint: Use compatibility settings to evolve schemas safely [OK]

Common Mistakes:

• Confusing encryption with compatibility
• Thinking rollback helps forward compatibility
• Disabling validation breaks schema safety