Schema evolution (backward, forward, full) in Kafka - Step-by-Step Execution

Process Flow - Schema evolution (backward, forward, full)

Start: Define initial schema

↓

Add new fields?

↓

Check backward compatibility

↓

Remove or rename fields?

↓

Check forward compatibility

↓

Both backward & forward?

↓

Full compatibility

↓

Schema accepted or rejected

Shows how schema changes are checked step-by-step for backward, forward, and full compatibility before acceptance.

Execution Sample

Kafka

Initial schema: {name:string}
Add field: age:int (default=0)
Check backward compatibility
Check forward compatibility
Result: Full compatibility

This example adds a new field with a default value and checks all compatibility types.

Process Table

Step	Schema Change	Backward Compatible?	Forward Compatible?	Full Compatible?	Result
1	Initial schema {name:string}	Yes (base)	Yes (base)	Yes (base)	Schema accepted
2	Add field age:int with default=0	Yes (old data valid)	Yes (new data valid)	Yes	Schema accepted
3	Remove field name	No (old data missing)	Yes	No	Schema rejected
4	Rename field name to fullname	No (field missing)	No (field missing)	No	Schema rejected
5	Add field email:string no default	No (old data missing field)	Yes	No	Schema rejected

💡 Schema changes are accepted only if compatibility checks pass according to the evolution type.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5
Schema Fields	{name:string}	{name:string, age:int=0}	{age:int=0}	{fullname:string}	{fullname:string, email:string}
Backward Compatible	Yes	Yes	No	No	No
Forward Compatible	Yes	Yes	Yes	No	Yes
Full Compatible	Yes	Yes	No	No	No

Key Moments - 3 Insights

Why is adding a new field with no default value not backward compatible?

Because old data does not have the new field, so deserialization fails without a default. See step 5 in execution_table where backward compatibility is No.

Why does renaming a field break both backward and forward compatibility?

Renaming removes the old field and adds a new one, so old data can't find the renamed field and new data can't find the old field. See step 4 in execution_table.

What does full compatibility require?

Both backward and forward compatibility must be true. See step 2 where full compatibility is Yes because both backward and forward are Yes.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What is the backward compatibility result?

A Yes

B No

C Partial

D Unknown

💡 Hint

Check the 'Backward Compatible?' column at step 3 in execution_table.

At which step does the schema become fully compatible again after the initial schema?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look for 'Full Compatible?' column with 'Yes' after step 1.

If you add a new field with a default value, how does it affect backward compatibility?

A It breaks backward compatibility

B It keeps backward compatibility

C It breaks forward compatibility

D It breaks full compatibility

💡 Hint

See step 2 in execution_table where adding a field with default keeps backward compatibility.

Concept Snapshot

Schema evolution rules:
- Backward compatible: new schema reads old data
- Forward compatible: old schema reads new data
- Full compatible: both backward and forward
- Adding fields with defaults is safe
- Removing or renaming fields breaks compatibility

Full Transcript

Schema evolution in Kafka means changing data formats without breaking existing data or applications. Backward compatibility means new schema can read old data. Forward compatibility means old schema can read new data. Full compatibility means both backward and forward are true. Adding new fields with default values keeps compatibility. Removing or renaming fields usually breaks it. This visual shows step-by-step how schema changes affect compatibility and when schemas are accepted or rejected.

Practice

1. What does backward compatibility mean in Kafka schema evolution?

easy

A. New consumers can read data produced with the old schema.

B. Old consumers can read data produced with the new schema.

C. Both old and new consumers can read all data versions.

D. Schemas cannot be changed once set.

Solution

1. Step 1: Understand backward compatibility meaning

   Backward compatibility means older consumers can still read data written with a newer schema.

2. Step 2: Match definition to options

   Old consumers can read data produced with the new schema. states old consumers can read new data, which matches backward compatibility.

3. Final Answer:

   Old consumers can read data produced with the new schema. -> Option B

4. Quick Check:

   Backward compatibility = old consumers read new data [OK]

Hint: Backward means old consumers read new data [OK]

Common Mistakes:

• Confusing backward with forward compatibility
• Thinking backward means new consumers read old data
• Assuming schemas cannot change

2. Which of the following is the correct way to declare a new optional field with a default value in an Avro schema for forward compatibility?

easy

A. {"name": "age", "type": "string", "default": "unknown"}

B. {"name": "age", "type": "int", "default": null}

C. {"name": "age", "type": "int", "default": 0}

D. {"name": "age", "type": "int"}

Solution

1. Step 1: Identify forward compatibility rule

   Forward compatibility requires new fields to have default values so new consumers can read data produced with the old schema.

2. Step 2: Check options for default value and type

   {"name": "age", "type": "int", "default": 0} declares an int field with default 0, which is correct for forward compatibility.

3. Final Answer:

   {"name": "age", "type": "int", "default": 0} -> Option C

4. Quick Check:

   New optional field with default = {"name": "age", "type": "int", "default": 0} [OK]

Hint: New fields need defaults for forward compatibility [OK]

Common Mistakes:

• Missing default value for new fields
• Using null default without union type
• Setting wrong type for default

3. Given this Avro schema evolution scenario:

Old schema: {"name": "id", "type": "int"}
New schema: {"name": "id", "type": "int"}, {"name": "name", "type": "string", "default": "unknown"}

What compatibility type does this change support?

medium

A. Backward compatible only

B. Forward compatible only

C. Not compatible

D. Full compatible

Solution

1. Step 1: Analyze schema change

   A new field "name" with a default value "unknown" is added to the schema.

2. Step 2: Determine compatibility

   Adding a field with a default supports both backward and forward compatibility, so it is full compatible.

3. Final Answer:

   Full compatible -> Option D

4. Quick Check:

   Adding field with default = full compatibility [OK]

Hint: Adding defaulted fields means full compatibility [OK]

Common Mistakes:

• Thinking adding fields breaks backward compatibility
• Ignoring default values
• Confusing forward and backward compatibility

4. You have this Avro schema change:

Old schema: {"name": "email", "type": "string"}
New schema: {"name": "email", "type": ["null", "string"], "default": null}

What is the main issue preventing backward compatibility?

medium

A. Changing field type from string to union breaks old consumers.

B. Adding default null makes schema incompatible.

C. Removing the field causes backward incompatibility.

D. No issue; schema is backward compatible.

Solution

1. Step 1: Identify schema change impact

   The field type changed from string to a union of null and string, which is a type change.

2. Step 2: Understand backward compatibility rules

   Changing a field's type breaks backward compatibility because old consumers expect a string, not a union.

3. Final Answer:

   Changing field type from string to union breaks old consumers. -> Option A

4. Quick Check:

   Type change breaks backward compatibility [OK]

Hint: Changing field type breaks backward compatibility [OK]

Common Mistakes:

• Assuming default null causes incompatibility
• Thinking adding union is always safe
• Ignoring type changes impact

5. You want to evolve a Kafka Avro schema to add a new field status that must be backward and forward compatible. Which approach is best?

hard

A. Add status as an optional string field with default value "active".

B. Add status as a required string field without default.

C. Remove an old optional field and add status as required.

D. Change an existing field type to string and add status without default.

Solution

1. Step 1: Understand compatibility requirements

   Backward and forward compatibility require new fields to have default values and be optional.

2. Step 2: Evaluate options

   Add status as an optional string field with default value "active". adds status as optional with default "active", satisfying both compatibilities.

3. Final Answer:

   Add status as an optional string field with default value "active". -> Option A

4. Quick Check:

   Optional field with default ensures full compatibility [OK]

Hint: New fields need defaults for full compatibility [OK]

Common Mistakes:

• Adding required fields without defaults
• Removing fields breaks compatibility
• Changing existing field types