Schema compatibility rules in Kafka - Step-by-Step Execution

Process Flow - Schema compatibility rules

Start with Existing Schema

↓

New Schema Submitted

↓

Check Compatibility Type

↓

BACKWARD

↓

Compare New Schema with Old Schema

↓

Is Compatibility Maintained?

↓

Accept New Schema

↓

End

The flow checks if a new schema is compatible with the existing one based on the chosen compatibility rule, then accepts or rejects it.

Execution Sample

Kafka

Existing schema: {"type":"record","fields":[{"name":"id","type":"int"}]}
New schema: {"type":"record","fields":[{"name":"id","type":"int"},{"name":"name","type":"string","default":""}]}
Compatibility: BACKWARD
Result: Accepted

This example shows adding a new optional field with a default value under backward compatibility, which is accepted.

Process Table

Step	Action	Compatibility Type	Check Description	Result
1	Start with existing schema	-	-	Ready for new schema
2	Submit new schema	-	-	New schema received
3	Check compatibility type	BACKWARD	New schema must read old data	Proceed to check
4	Compare schemas	BACKWARD	New schema adds field with default	Compatible
5	Decision	BACKWARD	Compatibility maintained	Accept new schema
6	End	-	-	Schema updated successfully

💡 New schema accepted because it is backward compatible with the existing schema.

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
existing_schema	{"id": "int"}	{"id": "int"}	{"id": "int"}	{"id": "int"}
new_schema	N/A	{"id": "int", "name": "string" (default)}	{"id": "int", "name": "string" (default)}	{"id": "int", "name": "string" (default)}
compatibility_type	N/A	BACKWARD	BACKWARD	BACKWARD
compatibility_result	N/A	N/A	Compatible	Accepted

Key Moments - 3 Insights

Why is adding a new field with a default value backward compatible?

Because old data without the new field can still be read by the new schema using the default value, as shown in step 4 of the execution_table.

What happens if the new schema removes a field under backward compatibility?

It breaks backward compatibility because the new schema cannot read old data that contains the removed field, leading to rejection as per the compatibility check.

How does full compatibility differ from backward or forward?

Full compatibility requires both backward and forward compatibility, meaning new and old schemas can read each other's data, ensuring maximum safety.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what compatibility check is performed?

A Backward compatibility check

B Forward compatibility check

C Full compatibility check

D No compatibility check

💡 Hint

Refer to the 'Compatibility Type' column at step 4 in the execution_table.

At which step is the new schema accepted?

A Step 3

B Step 4

C Step 5

D Step 6

💡 Hint

Check the 'Result' column for acceptance in the execution_table.

If the new schema removed a required field, what would change in the execution_table?

A Step 4 would show 'Compatible'

B Step 5 would show 'Reject new schema'

C Step 3 would change compatibility type

D Step 6 would be skipped

💡 Hint

Look at the 'Check Description' and 'Result' columns in steps 4 and 5.

Concept Snapshot

Schema compatibility rules ensure new schemas work with old data.
Backward: new schema reads old data.
Forward: old schema reads new data.
Full: both backward and forward.
Adding fields with defaults is backward compatible.
Removing fields breaks backward compatibility.

Full Transcript

Schema compatibility rules in Kafka control how new schemas relate to existing ones. When a new schema is submitted, the system checks compatibility based on the chosen rule: backward, forward, or full. Backward compatibility means the new schema can read data written with the old schema. Forward compatibility means the old schema can read data written with the new schema. Full compatibility requires both backward and forward compatibility. For example, adding a new field with a default value is backward compatible because old data without that field can still be read using the default. Removing a field breaks backward compatibility because the new schema cannot read old data containing that field. The process accepts or rejects the new schema accordingly.