Schema evolution (backward, forward, full) in Kafka - Commands & Configuration

Introduction

When you change the structure of data sent between systems, you need a way to keep old and new versions working together. Schema evolution helps manage these changes safely so your apps keep talking without breaking.

When you add a new field to a message but want old consumers to still read the data without errors

When you remove a field from a message and want new consumers to handle old messages gracefully

When you want to allow both old and new versions of a message schema to coexist during a gradual upgrade

When you want to prevent breaking changes that cause message deserialization failures

When you want to enforce compatibility rules automatically in your Kafka schema registry

Config File - schema-registry-config.properties

schema-registry-config.properties

schema.registry.url=http://localhost:8081
compatibility.level=BACKWARD

This configuration file sets the URL for the Kafka Schema Registry service and defines the compatibility level for schema evolution.

schema.registry.url: The address where the schema registry listens.

compatibility.level: Defines how new schemas can evolve compared to old ones. Options include BACKWARD, FORWARD, FULL, or NONE.

Commands

This command registers the initial schema for the 'user-value' subject in the schema registry. It defines a record with one field 'name' of type string.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"}' http://localhost:8081/subjects/user-value/versions

Expected Output

{"id":1}

This command registers a new version of the schema adding an optional 'age' field with a default null value. This is a backward compatible change.

Terminal

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"default\":null}]}"}' http://localhost:8081/subjects/user-value/versions

Expected Output

{"id":2}

This command retrieves the current compatibility setting for the 'user-value' subject to verify the schema evolution policy.

Terminal

curl http://localhost:8081/config/user-value

Expected Output

{"compatibility":"BACKWARD"}

This command updates the compatibility level for the 'user-value' subject to FULL, allowing both backward and forward compatibility.

Terminal

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility": "FULL"}' http://localhost:8081/config/user-value

Expected Output

{"compatibility":"FULL"}

Key Concept

If you remember nothing else from this pattern, remember: schema evolution rules ensure new and old message versions work together without breaking your Kafka data flow.

Common Mistakes

Registering a new schema version that removes a required field without a default value

This breaks backward compatibility and causes consumers expecting the old schema to fail when reading messages.

Add default values for removed fields or use optional types to maintain compatibility.

Not setting the compatibility level in the schema registry

Without compatibility enforcement, incompatible schema changes can be registered, causing runtime errors.

Always configure compatibility level (BACKWARD, FORWARD, or FULL) to prevent breaking changes.

Assuming adding a new field without a default is backward compatible

Consumers using the old schema will fail to read messages missing the new field's value.

Add new fields as optional with default values to keep backward compatibility.

Summary

Register schemas in the Kafka Schema Registry to define message structure versions.

Set compatibility levels to control how schemas can evolve without breaking consumers.

Use backward, forward, or full compatibility depending on your upgrade strategy.

Verify and update compatibility settings using schema registry REST API commands.

Practice

1. What does backward compatibility mean in Kafka schema evolution?

easy

A. New consumers can read data produced with the old schema.

B. Old consumers can read data produced with the new schema.

C. Both old and new consumers can read all data versions.

D. Schemas cannot be changed once set.

Solution

1. Step 1: Understand backward compatibility meaning

   Backward compatibility means older consumers can still read data written with a newer schema.

2. Step 2: Match definition to options

   Old consumers can read data produced with the new schema. states old consumers can read new data, which matches backward compatibility.

3. Final Answer:

   Old consumers can read data produced with the new schema. -> Option B

4. Quick Check:

   Backward compatibility = old consumers read new data [OK]

Hint: Backward means old consumers read new data [OK]

Common Mistakes:

• Confusing backward with forward compatibility
• Thinking backward means new consumers read old data
• Assuming schemas cannot change

2. Which of the following is the correct way to declare a new optional field with a default value in an Avro schema for forward compatibility?

easy

A. {"name": "age", "type": "string", "default": "unknown"}

B. {"name": "age", "type": "int", "default": null}

C. {"name": "age", "type": "int", "default": 0}

D. {"name": "age", "type": "int"}

Solution

1. Step 1: Identify forward compatibility rule

   Forward compatibility requires new fields to have default values so new consumers can read data produced with the old schema.

2. Step 2: Check options for default value and type

   {"name": "age", "type": "int", "default": 0} declares an int field with default 0, which is correct for forward compatibility.

3. Final Answer:

   {"name": "age", "type": "int", "default": 0} -> Option C

4. Quick Check:

   New optional field with default = {"name": "age", "type": "int", "default": 0} [OK]

Hint: New fields need defaults for forward compatibility [OK]

Common Mistakes:

• Missing default value for new fields
• Using null default without union type
• Setting wrong type for default

3. Given this Avro schema evolution scenario:

Old schema: {"name": "id", "type": "int"}
New schema: {"name": "id", "type": "int"}, {"name": "name", "type": "string", "default": "unknown"}

What compatibility type does this change support?

medium

A. Backward compatible only

B. Forward compatible only

C. Not compatible

D. Full compatible

Solution

1. Step 1: Analyze schema change

   A new field "name" with a default value "unknown" is added to the schema.

2. Step 2: Determine compatibility

   Adding a field with a default supports both backward and forward compatibility, so it is full compatible.

3. Final Answer:

   Full compatible -> Option D

4. Quick Check:

   Adding field with default = full compatibility [OK]

Hint: Adding defaulted fields means full compatibility [OK]

Common Mistakes:

• Thinking adding fields breaks backward compatibility
• Ignoring default values
• Confusing forward and backward compatibility

4. You have this Avro schema change:

Old schema: {"name": "email", "type": "string"}
New schema: {"name": "email", "type": ["null", "string"], "default": null}

What is the main issue preventing backward compatibility?

medium

A. Changing field type from string to union breaks old consumers.

B. Adding default null makes schema incompatible.

C. Removing the field causes backward incompatibility.

D. No issue; schema is backward compatible.

Solution

1. Step 1: Identify schema change impact

   The field type changed from string to a union of null and string, which is a type change.

2. Step 2: Understand backward compatibility rules

   Changing a field's type breaks backward compatibility because old consumers expect a string, not a union.

3. Final Answer:

   Changing field type from string to union breaks old consumers. -> Option A

4. Quick Check:

   Type change breaks backward compatibility [OK]

Hint: Changing field type breaks backward compatibility [OK]

Common Mistakes:

• Assuming default null causes incompatibility
• Thinking adding union is always safe
• Ignoring type changes impact

5. You want to evolve a Kafka Avro schema to add a new field status that must be backward and forward compatible. Which approach is best?

hard

A. Add status as an optional string field with default value "active".

B. Add status as a required string field without default.

C. Remove an old optional field and add status as required.

D. Change an existing field type to string and add status without default.

Solution

1. Step 1: Understand compatibility requirements

   Backward and forward compatibility require new fields to have default values and be optional.

2. Step 2: Evaluate options

   Add status as an optional string field with default value "active". adds status as optional with default "active", satisfying both compatibilities.

3. Final Answer:

   Add status as an optional string field with default value "active". -> Option A

4. Quick Check:

   Optional field with default ensures full compatibility [OK]

Hint: New fields need defaults for full compatibility [OK]

Common Mistakes:

• Adding required fields without defaults
• Removing fields breaks compatibility
• Changing existing field types