Schema evolution (backward, forward, full) in Kafka - Commands & Configuration

Introduction

When you change the structure of data sent between systems, you need a way to keep old and new versions working together. Schema evolution helps manage these changes safely so your apps keep talking without breaking.

When you add a new field to a message but want old consumers to still read the data without errors

When you remove a field from a message and want new consumers to handle old messages gracefully

When you want to allow both old and new versions of a message schema to coexist during a gradual upgrade

When you want to prevent breaking changes that cause message deserialization failures

When you want to enforce compatibility rules automatically in your Kafka schema registry

Config File - schema-registry-config.properties

schema-registry-config.properties

schema.registry.url=http://localhost:8081
compatibility.level=BACKWARD

This configuration file sets the URL for the Kafka Schema Registry service and defines the compatibility level for schema evolution.

schema.registry.url: The address where the schema registry listens.

compatibility.level: Defines how new schemas can evolve compared to old ones. Options include BACKWARD, FORWARD, FULL, or NONE.

Commands

This command registers the initial schema for the 'user-value' subject in the schema registry. It defines a record with one field 'name' of type string.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/user-value/versions

Expected Output

{"id":1}

This command registers a new version of the schema adding an optional 'age' field with a default null value. This is a backward compatible change.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"default\":null}]}"}' http://localhost:8081/subjects/user-value/versions

Expected Output

{"id":2}

This command retrieves the current compatibility setting for the 'user-value' subject to verify the schema evolution policy.

Terminal

curl http://localhost:8081/config/user-value

Expected Output

{"compatibility":"BACKWARD"}

This command updates the compatibility level for the 'user-value' subject to FULL, allowing both backward and forward compatibility.

Terminal

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility": "FULL"}' http://localhost:8081/config/user-value

Expected Output

{"compatibility":"FULL"}

Key Concept

If you remember nothing else from this pattern, remember: schema evolution rules ensure new and old message versions work together without breaking your Kafka data flow.

Common Mistakes

Registering a new schema version that removes a required field without a default value

This breaks backward compatibility and causes consumers expecting the old schema to fail when reading messages.

Add default values for removed fields or use optional types to maintain compatibility.

Not setting the compatibility level in the schema registry

Without compatibility enforcement, incompatible schema changes can be registered, causing runtime errors.

Always configure compatibility level (BACKWARD, FORWARD, or FULL) to prevent breaking changes.

Assuming adding a new field without a default is backward compatible

Consumers using the old schema will fail to read messages missing the new field's value.

Add new fields as optional with default values to keep backward compatibility.

Summary

Register schemas in the Kafka Schema Registry to define message structure versions.

Set compatibility levels to control how schemas can evolve without breaking consumers.

Use backward, forward, or full compatibility depending on your upgrade strategy.

Verify and update compatibility settings using schema registry REST API commands.