Why Schema evolution (backward, forward, full) in Kafka? - Purpose & Use Cases

The Big Idea

What if you could change your data format without breaking anything anywhere?

The Scenario

Imagine you have a big box of recipe cards that you share with friends. Every time you change a recipe, you rewrite all the cards from scratch and send them again. Your friends get confused because some cards are missing ingredients or steps they expect.

The Problem

Manually updating and sharing data formats is slow and risky. If you change something, old programs might break because they expect the old format. New programs might miss data if they only understand the new format. This causes errors and confusion.

The Solution

Schema evolution lets you change data formats smoothly. It defines rules for adding or removing parts so old and new programs can still understand each other. This means you can update your data without breaking anything.

Before vs After

✗ Before

Producer sends data with old format
Consumer expects old format
Change format breaks consumer

✓ After

Define schema with evolution rules
Producer and consumer adapt automatically
Data stays compatible

What It Enables

It enables seamless data updates where old and new systems work together without errors.

Real Life Example

A company updates its customer data format to add a new phone number field. Thanks to schema evolution, old services still read data without crashing, and new services use the new field smoothly.

Key Takeaways

Manual data format changes cause errors and confusion.

Schema evolution defines safe rules for changing data formats.

This keeps old and new systems working together smoothly.

Practice

1. What does backward compatibility mean in Kafka schema evolution?

easy

A. New consumers can read data produced with the old schema.

B. Old consumers can read data produced with the new schema.

C. Both old and new consumers can read all data versions.

D. Schemas cannot be changed once set.

Solution

1. Step 1: Understand backward compatibility meaning

   Backward compatibility means older consumers can still read data written with a newer schema.

2. Step 2: Match definition to options

   Old consumers can read data produced with the new schema. states old consumers can read new data, which matches backward compatibility.

3. Final Answer:

   Old consumers can read data produced with the new schema. -> Option B

4. Quick Check:

   Backward compatibility = old consumers read new data [OK]

Hint: Backward means old consumers read new data [OK]

Common Mistakes:

• Confusing backward with forward compatibility
• Thinking backward means new consumers read old data
• Assuming schemas cannot change

2. Which of the following is the correct way to declare a new optional field with a default value in an Avro schema for forward compatibility?

easy

A. {"name": "age", "type": "string", "default": "unknown"}

B. {"name": "age", "type": "int", "default": null}

C. {"name": "age", "type": "int", "default": 0}

D. {"name": "age", "type": "int"}

Solution

1. Step 1: Identify forward compatibility rule

   Forward compatibility requires new fields to have default values so new consumers can read data produced with the old schema.

2. Step 2: Check options for default value and type

   {"name": "age", "type": "int", "default": 0} declares an int field with default 0, which is correct for forward compatibility.

3. Final Answer:

   {"name": "age", "type": "int", "default": 0} -> Option C

4. Quick Check:

   New optional field with default = {"name": "age", "type": "int", "default": 0} [OK]

Hint: New fields need defaults for forward compatibility [OK]

Common Mistakes:

• Missing default value for new fields
• Using null default without union type
• Setting wrong type for default

3. Given this Avro schema evolution scenario:

Old schema: {"name": "id", "type": "int"}
New schema: {"name": "id", "type": "int"}, {"name": "name", "type": "string", "default": "unknown"}

What compatibility type does this change support?

medium

A. Backward compatible only

B. Forward compatible only

C. Not compatible

D. Full compatible

Solution

1. Step 1: Analyze schema change

   A new field "name" with a default value "unknown" is added to the schema.

2. Step 2: Determine compatibility

   Adding a field with a default supports both backward and forward compatibility, so it is full compatible.

3. Final Answer:

   Full compatible -> Option D

4. Quick Check:

   Adding field with default = full compatibility [OK]

Hint: Adding defaulted fields means full compatibility [OK]

Common Mistakes:

• Thinking adding fields breaks backward compatibility
• Ignoring default values
• Confusing forward and backward compatibility

4. You have this Avro schema change:

Old schema: {"name": "email", "type": "string"}
New schema: {"name": "email", "type": ["null", "string"], "default": null}

What is the main issue preventing backward compatibility?

medium

A. Changing field type from string to union breaks old consumers.

B. Adding default null makes schema incompatible.

C. Removing the field causes backward incompatibility.

D. No issue; schema is backward compatible.

Solution

1. Step 1: Identify schema change impact

   The field type changed from string to a union of null and string, which is a type change.

2. Step 2: Understand backward compatibility rules

   Changing a field's type breaks backward compatibility because old consumers expect a string, not a union.

3. Final Answer:

   Changing field type from string to union breaks old consumers. -> Option A

4. Quick Check:

   Type change breaks backward compatibility [OK]

Hint: Changing field type breaks backward compatibility [OK]

Common Mistakes:

• Assuming default null causes incompatibility
• Thinking adding union is always safe
• Ignoring type changes impact

5. You want to evolve a Kafka Avro schema to add a new field status that must be backward and forward compatible. Which approach is best?

hard

A. Add status as an optional string field with default value "active".

B. Add status as a required string field without default.

C. Remove an old optional field and add status as required.

D. Change an existing field type to string and add status without default.

Solution

1. Step 1: Understand compatibility requirements

   Backward and forward compatibility require new fields to have default values and be optional.

2. Step 2: Evaluate options

   Add status as an optional string field with default value "active". adds status as optional with default "active", satisfying both compatibilities.

3. Final Answer:

   Add status as an optional string field with default value "active". -> Option A

4. Quick Check:

   Optional field with default ensures full compatibility [OK]

Hint: New fields need defaults for full compatibility [OK]

Common Mistakes:

• Adding required fields without defaults
• Removing fields breaks compatibility
• Changing existing field types