Schema Registry concept in Kafka - Commands & Configuration

Introduction

When different applications send and receive data, they need to agree on the data format. Schema Registry helps by storing and managing these data formats so apps can understand each other without confusion.

When you want to ensure all messages in Kafka follow a consistent format.

When multiple teams produce or consume data and need a shared understanding of data structure.

When you want to evolve data formats safely without breaking existing consumers.

When you want to validate data before it is sent to Kafka to avoid errors.

When you want to track versions of data formats over time.

Config File - schema-registry.properties

schema-registry.properties

listeners=http://0.0.0.0:8081
kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
kafkastore.topic=_schemas
debug=false

listeners: Defines where the Schema Registry service listens for requests.

kafkastore.bootstrap.servers: Kafka server address where schemas are stored.

kafkastore.topic: Kafka topic used internally to store schemas.

debug: Enables or disables debug logging.

Commands

Starts the Schema Registry service using the configuration file to manage schemas centrally.

Terminal

schema-registry-start schema-registry.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Starting Schema Registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain) 

[2024-06-01 12:00:01,000] INFO Schema Registry started and listening at http://0.0.0.0:8081 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)

Registers a new schema for the subject 'User-value' in the Schema Registry so producers and consumers can use it.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/User-value/versions

Expected Output

{"id":1}

Lists all subjects (data formats) currently registered in the Schema Registry.

Terminal

curl http://localhost:8081/subjects

Expected Output

["User-value"]

Retrieves the schema details for version 1 of the 'User-value' subject to verify its structure.

Terminal

curl http://localhost:8081/subjects/User-value/versions/1

Expected Output

{"subject":"User-value","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}

Key Concept

If you remember nothing else from this pattern, remember: Schema Registry stores and manages data formats so all apps agree on how data looks.

Common Mistakes

Not starting the Schema Registry service before registering schemas.

Schema Registry commands fail because the service is not running to accept requests.

Always start the Schema Registry service first using the configuration file.

Registering schemas with invalid JSON or missing escape characters.

Schema registration fails due to malformed JSON, causing errors in data format understanding.

Ensure the schema JSON is valid and properly escaped when sending via curl.

Not using the correct content-type header when registering schemas.

Schema Registry rejects requests without the proper content-type, so schemas are not saved.

Use 'Content-Type: application/vnd.schemaregistry.v1+json' header in schema registration requests.

Summary

Start the Schema Registry service with a configuration file to manage schemas centrally.

Register schemas via HTTP POST requests to define data formats for Kafka topics.

Use HTTP GET requests to list and retrieve schema versions for validation and compatibility.

Practice

1. What is the main purpose of a Schema Registry in Kafka?

easy

A. To encrypt Kafka messages during transmission

B. To monitor Kafka broker performance

C. To route messages between Kafka topics

D. To store and manage data schemas for Kafka messages

Solution

1. Step 1: Understand the role of Schema Registry

   Schema Registry stores and manages the structure (schema) of data used in Kafka messages.

2. Step 2: Compare with other Kafka components

   Monitoring, routing, and encryption are handled by other Kafka tools, not Schema Registry.

3. Final Answer:

   To store and manage data schemas for Kafka messages -> Option D

4. Quick Check:

   Schema Registry = data schema storage [OK]

Hint: Schema Registry manages data formats, not Kafka internals [OK]

Common Mistakes:

• Confusing Schema Registry with Kafka brokers
• Thinking it handles message routing
• Assuming it encrypts messages

2. Which of the following is the correct way to register a schema using Kafka Schema Registry API?

easy

A. DELETE /topics/{topic}/schemas with schema in YAML

B. GET /schemas/{id} with schema in XML

C. POST /subjects/{subject}/versions with schema in JSON

D. PUT /brokers/{brokerId}/schemas with schema in CSV

Solution

1. Step 1: Identify the correct HTTP method and endpoint

   Registering a schema uses POST on /subjects/{subject}/versions with JSON schema.

2. Step 2: Check other options for correctness

   GET retrieves schemas, DELETE removes topics (not schemas), PUT on brokers is invalid for schema registration.

3. Final Answer:

   POST /subjects/{subject}/versions with schema in JSON -> Option C

4. Quick Check:

   Register schema = POST with JSON [OK]

Hint: Register schemas with POST and JSON format [OK]

Common Mistakes:

• Using GET instead of POST to register
• Confusing schema formats (JSON vs XML/YAML/CSV)
• Using wrong API endpoints

3. Given this code snippet using Kafka Schema Registry client, what will be the output?

schema = '{"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
print(schema_id)

medium

A. An integer ID representing the registered schema

B. The original schema string printed

C. A JSON object with schema details

D. A runtime error due to missing client method

Solution

1. Step 1: Understand the register_schema method

   This method registers the schema and returns an integer ID assigned by Schema Registry.

2. Step 2: Analyze the print statement

   Printing schema_id outputs the integer ID, not the schema string or JSON object.

3. Final Answer:

   An integer ID representing the registered schema -> Option A

4. Quick Check:

   Register returns schema ID integer [OK]

Hint: Register returns schema ID number, not schema text [OK]

Common Mistakes:

• Expecting schema string instead of ID
• Confusing output with JSON object
• Assuming method does not exist

4. Identify the error in this Kafka Schema Registry usage snippet:

schema = '{"type":"record","name":"User","fields":[{"name":"age","type":"int"}]}'
subject = "User-value"
schema_id = client.register_schema(subject, schema)
client.register_schema(subject, schema)

medium

A. Registering the same schema twice causes an error

B. No error; registering same schema returns same ID

C. Schema string is not valid JSON

D. Missing schema version in registration call

Solution

1. Step 1: Check schema validity

   The schema string is valid JSON describing a record with an int field.

2. Step 2: Understand Schema Registry behavior on duplicate schemas

   Registering the same schema again returns the same schema ID without error.

3. Final Answer:

   No error; registering same schema returns same ID -> Option B

4. Quick Check:

   Duplicate schema registration returns same ID [OK]

Hint: Duplicate schema registration is safe and returns same ID [OK]

Common Mistakes:

• Assuming duplicate registration causes error
• Thinking schema string is invalid JSON
• Expecting version parameter in register call

5. You want to evolve a Kafka message schema by adding a new optional field without breaking existing consumers. Which Schema Registry feature helps achieve this safely?

hard

A. Schema compatibility settings (e.g., BACKWARD compatibility)

B. Schema encryption for secure transmission

C. Schema version rollback to previous schema

D. Schema validation disabling for faster registration

Solution

1. Step 1: Understand schema evolution needs

   Adding a new optional field requires compatibility so old consumers still work.

2. Step 2: Identify Schema Registry feature for compatibility

   Schema compatibility settings like BACKWARD ensure new schemas don't break old consumers.

3. Step 3: Eliminate other options

   Encryption, rollback, or disabling validation do not help safe schema evolution.

4. Final Answer:

   Schema compatibility settings (e.g., BACKWARD compatibility) -> Option A

5. Quick Check:

   Compatibility settings enable safe schema evolution [OK]

Hint: Use compatibility settings to evolve schemas safely [OK]

Common Mistakes:

• Confusing encryption with compatibility
• Thinking rollback helps forward compatibility
• Disabling validation breaks schema safety