Schema Registry concept in Kafka - Commands & Configuration

Introduction

When different applications send and receive data, they need to agree on the data format. Schema Registry helps by storing and managing these data formats so apps can understand each other without confusion.

When you want to ensure all messages in Kafka follow a consistent format.

When multiple teams produce or consume data and need a shared understanding of data structure.

When you want to evolve data formats safely without breaking existing consumers.

When you want to validate data before it is sent to Kafka to avoid errors.

When you want to track versions of data formats over time.

Config File - schema-registry.properties

schema-registry.properties

listeners=http://0.0.0.0:8081
kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
kafkastore.topic=_schemas
debug=false

listeners: Defines where the Schema Registry service listens for requests.

kafkastore.bootstrap.servers: Kafka server address where schemas are stored.

kafkastore.topic: Kafka topic used internally to store schemas.

debug: Enables or disables debug logging.

Commands

Starts the Schema Registry service using the configuration file to manage schemas centrally.

Terminal

schema-registry-start schema-registry.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Starting Schema Registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain) 

[2024-06-01 12:00:01,000] INFO Schema Registry started and listening at http://0.0.0.0:8081 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)

Registers a new schema for the subject 'User-value' in the Schema Registry so producers and consumers can use it.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/User-value/versions

Expected Output

{"id":1}

Lists all subjects (data formats) currently registered in the Schema Registry.

Terminal

curl http://localhost:8081/subjects

Expected Output

["User-value"]

Retrieves the schema details for version 1 of the 'User-value' subject to verify its structure.

Terminal

curl http://localhost:8081/subjects/User-value/versions/1

Expected Output

{"subject":"User-value","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}

Key Concept

If you remember nothing else from this pattern, remember: Schema Registry stores and manages data formats so all apps agree on how data looks.

Common Mistakes

Not starting the Schema Registry service before registering schemas.

Schema Registry commands fail because the service is not running to accept requests.

Always start the Schema Registry service first using the configuration file.

Registering schemas with invalid JSON or missing escape characters.

Schema registration fails due to malformed JSON, causing errors in data format understanding.

Ensure the schema JSON is valid and properly escaped when sending via curl.

Not using the correct content-type header when registering schemas.

Schema Registry rejects requests without the proper content-type, so schemas are not saved.

Use 'Content-Type: application/vnd.schemaregistry.v1+json' header in schema registration requests.

Summary

Start the Schema Registry service with a configuration file to manage schemas centrally.

Register schemas via HTTP POST requests to define data formats for Kafka topics.

Use HTTP GET requests to list and retrieve schema versions for validation and compatibility.