Schema evolution (backward, forward, full) in Kafka - Mini Project: Build & Apply

Schema Evolution with Kafka

📖 Scenario: You work at a company that uses Kafka to send user data between services. Over time, the data format changes. You need to handle these changes safely using schema evolution.

🎯 Goal: Build a simple Kafka producer and consumer that demonstrate backward, forward, and full schema evolution using Avro schemas.

📋 What You'll Learn

Create an initial Avro schema for user data

Add a new optional field to the schema (backward compatible)

Add a new required field with a default value (forward compatible)

Combine changes to support full compatibility

Produce and consume messages using the evolved schemas

💡 Why This Matters

🌍 Real World

Kafka is widely used for streaming data between services. Schema evolution lets you change data formats safely without breaking consumers or producers.

💼 Career

Understanding schema evolution is important for data engineers and backend developers working with Kafka, Avro, and schema registries to maintain reliable data pipelines.

1

Create the initial Avro schema

Create a string variable called initial_schema that holds this exact Avro schema JSON string: {"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}

Kafka

# Create the initial Avro schema string variable 'initial_schema'
# Your code here

Hint

Use a string variable named initial_schema and assign the exact JSON schema string.

2

Add a new optional field (backward compatible)

Create a string variable called backward_schema that adds a new optional field email of type ["null", "string"] with default null to the initial_schema. Use the same record name User and keep the other fields unchanged.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
# Create 'backward_schema' string with new optional field 'email'
# Your code here

Hint

Add the new field email as optional by using a union type with null and a default null.

3

Add a new required field with default (forward compatible)

Create a string variable called forward_schema that adds a new required field country of type string with default value "USA" to the initial_schema. Keep the record name User and other fields unchanged.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
backward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "email", "type": ["null", "string"], "default": null}]}'
# Create 'forward_schema' string with new required field 'country' with default "USA"
# Your code here

Hint

Add the new field country as required but provide a default value to keep forward compatibility.

4

Combine changes for full compatibility and print schemas

Create a string variable called full_schema that combines the optional email field and the required country field with default "USA" in the same schema. Then print the variables initial_schema, backward_schema, forward_schema, and full_schema each on its own line.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
backward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "email", "type": ["null", "string"], "default": null}]}'
forward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "country", "type": "string", "default": "USA"}]}'
# Create 'full_schema' string combining 'email' and 'country' fields
# Print all four schema variables each on its own line
# Your code here

Hint

Combine both new fields in full_schema and print all schemas exactly as strings.

Practice

1. What does backward compatibility mean in Kafka schema evolution?

easy

A. New consumers can read data produced with the old schema.

B. Old consumers can read data produced with the new schema.

C. Both old and new consumers can read all data versions.

D. Schemas cannot be changed once set.

Solution

1. Step 1: Understand backward compatibility meaning

   Backward compatibility means older consumers can still read data written with a newer schema.

2. Step 2: Match definition to options

   Old consumers can read data produced with the new schema. states old consumers can read new data, which matches backward compatibility.

3. Final Answer:

   Old consumers can read data produced with the new schema. -> Option B

4. Quick Check:

   Backward compatibility = old consumers read new data [OK]

Hint: Backward means old consumers read new data [OK]

Common Mistakes:

• Confusing backward with forward compatibility
• Thinking backward means new consumers read old data
• Assuming schemas cannot change

2. Which of the following is the correct way to declare a new optional field with a default value in an Avro schema for forward compatibility?

easy

A. {"name": "age", "type": "string", "default": "unknown"}

B. {"name": "age", "type": "int", "default": null}

C. {"name": "age", "type": "int", "default": 0}

D. {"name": "age", "type": "int"}

Solution

1. Step 1: Identify forward compatibility rule

   Forward compatibility requires new fields to have default values so new consumers can read data produced with the old schema.

2. Step 2: Check options for default value and type

   {"name": "age", "type": "int", "default": 0} declares an int field with default 0, which is correct for forward compatibility.

3. Final Answer:

   {"name": "age", "type": "int", "default": 0} -> Option C

4. Quick Check:

   New optional field with default = {"name": "age", "type": "int", "default": 0} [OK]

Hint: New fields need defaults for forward compatibility [OK]

Common Mistakes:

• Missing default value for new fields
• Using null default without union type
• Setting wrong type for default

3. Given this Avro schema evolution scenario:

Old schema: {"name": "id", "type": "int"}
New schema: {"name": "id", "type": "int"}, {"name": "name", "type": "string", "default": "unknown"}

What compatibility type does this change support?

medium

A. Backward compatible only

B. Forward compatible only

C. Not compatible

D. Full compatible

Solution

1. Step 1: Analyze schema change

   A new field "name" with a default value "unknown" is added to the schema.

2. Step 2: Determine compatibility

   Adding a field with a default supports both backward and forward compatibility, so it is full compatible.

3. Final Answer:

   Full compatible -> Option D

4. Quick Check:

   Adding field with default = full compatibility [OK]

Hint: Adding defaulted fields means full compatibility [OK]

Common Mistakes:

• Thinking adding fields breaks backward compatibility
• Ignoring default values
• Confusing forward and backward compatibility

4. You have this Avro schema change:

Old schema: {"name": "email", "type": "string"}
New schema: {"name": "email", "type": ["null", "string"], "default": null}

What is the main issue preventing backward compatibility?

medium

A. Changing field type from string to union breaks old consumers.

B. Adding default null makes schema incompatible.

C. Removing the field causes backward incompatibility.

D. No issue; schema is backward compatible.

Solution

1. Step 1: Identify schema change impact

   The field type changed from string to a union of null and string, which is a type change.

2. Step 2: Understand backward compatibility rules

   Changing a field's type breaks backward compatibility because old consumers expect a string, not a union.

3. Final Answer:

   Changing field type from string to union breaks old consumers. -> Option A

4. Quick Check:

   Type change breaks backward compatibility [OK]

Hint: Changing field type breaks backward compatibility [OK]

Common Mistakes:

• Assuming default null causes incompatibility
• Thinking adding union is always safe
• Ignoring type changes impact

5. You want to evolve a Kafka Avro schema to add a new field status that must be backward and forward compatible. Which approach is best?

hard

A. Add status as an optional string field with default value "active".

B. Add status as a required string field without default.

C. Remove an old optional field and add status as required.

D. Change an existing field type to string and add status without default.

Solution

1. Step 1: Understand compatibility requirements

   Backward and forward compatibility require new fields to have default values and be optional.

2. Step 2: Evaluate options

   Add status as an optional string field with default value "active". adds status as optional with default "active", satisfying both compatibilities.

3. Final Answer:

   Add status as an optional string field with default value "active". -> Option A

4. Quick Check:

   Optional field with default ensures full compatibility [OK]

Hint: New fields need defaults for full compatibility [OK]

Common Mistakes:

• Adding required fields without defaults
• Removing fields breaks compatibility
• Changing existing field types