Schema compatibility rules in Kafka - Commands & Configuration

Introduction

When you use Kafka to send data, you often use schemas to define the data format. Schema compatibility rules help ensure that new versions of schemas work well with old data, so your apps don't break when you change data formats.

When you want to update your data format without breaking existing consumers.

When multiple teams produce and consume data on the same Kafka topic and need to agree on data structure changes.

When you want to enforce rules on how schemas evolve over time to avoid data errors.

When you want to safely add new fields to your messages without affecting old consumers.

When you want to prevent incompatible schema changes that could cause data processing failures.

Commands

This command sends a message to the Kafka topic 'example-topic' using an Avro schema that defines a record with a single string field 'name'. It shows how to produce data with a schema.

Terminal

kafka-avro-console-producer --broker-list localhost:9092 --topic example-topic --property value.schema='{"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}'

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker address to connect to.

→

--topic - Specifies the Kafka topic to send messages to.

→

--property value.schema - Defines the Avro schema for the message value.

This command sets the schema compatibility rule for the topic 'example-topic' to BACKWARD. This means new schemas must be compatible with previous schemas so old data can still be read.

Terminal

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility":"BACKWARD"}' http://localhost:8081/config/example-topic

Expected Output

{"compatibility":"BACKWARD"}

This command retrieves the current schema compatibility setting for the topic 'example-topic' to verify the rule is set correctly.

Terminal

curl -X GET http://localhost:8081/config/example-topic

Expected Output

{"compatibility":"BACKWARD"}

Key Concept

If you remember nothing else from schema compatibility rules, remember: they protect your data consumers by ensuring new schema versions work with old data.

Common Mistakes

Setting compatibility to NONE and then changing schemas freely.

This can cause consumers to fail because they expect data in a certain format that no longer matches.

Use a compatibility rule like BACKWARD or FORWARD to ensure safe schema evolution.

Not verifying the compatibility setting after changing it.

You might think the rule is applied but it is not, leading to unexpected schema conflicts.

Always check the compatibility setting with a GET request after updating it.

Summary

Use schema compatibility rules to control how schemas evolve in Kafka topics.

Set compatibility modes like BACKWARD to keep new schemas compatible with old data.

Verify compatibility settings with API calls to avoid breaking consumers.