Schema evolution (backward, forward, full) in Kafka - Mini Project: Build & Apply

Schema Evolution with Kafka

📖 Scenario: You work at a company that uses Kafka to send user data between services. Over time, the data format changes. You need to handle these changes safely using schema evolution.

🎯 Goal: Build a simple Kafka producer and consumer that demonstrate backward, forward, and full schema evolution using Avro schemas.

📋 What You'll Learn

Create an initial Avro schema for user data

Add a new optional field to the schema (backward compatible)

Add a new required field with a default value (forward compatible)

Combine changes to support full compatibility

Produce and consume messages using the evolved schemas

💡 Why This Matters

🌍 Real World

Kafka is widely used for streaming data between services. Schema evolution lets you change data formats safely without breaking consumers or producers.

💼 Career

Understanding schema evolution is important for data engineers and backend developers working with Kafka, Avro, and schema registries to maintain reliable data pipelines.

1

Create the initial Avro schema

Create a string variable called initial_schema that holds this exact Avro schema JSON string: {"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}

Kafka

# Create the initial Avro schema string variable 'initial_schema'
# Your code here

Need a hint?

Use a string variable named initial_schema and assign the exact JSON schema string.

2

Add a new optional field (backward compatible)

Create a string variable called backward_schema that adds a new optional field email of type ["null", "string"] with default null to the initial_schema. Use the same record name User and keep the other fields unchanged.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
# Create 'backward_schema' string with new optional field 'email'
# Your code here

Need a hint?

Add the new field email as optional by using a union type with null and a default null.

3

Add a new required field with default (forward compatible)

Create a string variable called forward_schema that adds a new required field country of type string with default value "USA" to the initial_schema. Keep the record name User and other fields unchanged.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
backward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "email", "type": ["null", "string"], "default": null}]}'
# Create 'forward_schema' string with new required field 'country' with default "USA"
# Your code here

Need a hint?

Add the new field country as required but provide a default value to keep forward compatibility.

4

Combine changes for full compatibility and print schemas

Create a string variable called full_schema that combines the optional email field and the required country field with default "USA" in the same schema. Then print the variables initial_schema, backward_schema, forward_schema, and full_schema each on its own line.

Kafka

initial_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}]}'
backward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "email", "type": ["null", "string"], "default": null}]}'
forward_schema = '{"type": "record", "name": "User", "fields": [{"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "country", "type": "string", "default": "USA"}]}'
# Create 'full_schema' string combining 'email' and 'country' fields
# Print all four schema variables each on its own line
# Your code here

Need a hint?

Combine both new fields in full_schema and print all schemas exactly as strings.