Schema evolution (backward, forward, full) in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output when a consumer with an older schema reads a message with a newer schema?

Consider a Kafka topic using Avro serialization with schema evolution enabled. The producer adds a new optional field to the schema. The consumer uses the older schema without this new field. What happens when the consumer reads the message?

Kafka

Producer schema v2: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":["null","int"],"default":null}]}
Consumer schema v1: {"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}

A The consumer reads the message but sets 'age' to zero by default.

B The consumer throws a serialization error due to the missing 'age' field.

C The consumer blocks and waits for a schema update.

D The consumer reads the message successfully, ignoring the new 'age' field.

💡 Hint

Think about forward compatibility: can an older consumer read data produced with the newer schema?

Forward compatibility means new schemas can add optional fields with defaults. Older consumers ignore these new fields and read data successfully.

❓ Predict Output

intermediate

What happens when a producer sends data with an older schema and the consumer expects a newer schema?

A Kafka consumer expects a schema with a new required field added (no default). The producer sends data using the older schema without this field. What is the result when the consumer tries to deserialize the message?

Kafka

Producer schema v1: {"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}
Consumer schema v2: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"email","type":"string"}]}

A The consumer throws a deserialization error due to the missing 'email' field.

B The consumer sets 'email' to an empty string by default.

C The consumer ignores the missing 'email' field and reads the message.

D The consumer reads the message but sets 'email' to null.

💡 Hint

Consider backward compatibility and required fields without defaults.

Backward compatibility requires that new schemas can read old data. But if a new required field has no default, the consumer cannot fill it, causing a deserialization error.

🧠 Conceptual

advanced

Which schema evolution mode allows both backward and forward compatibility?

In Kafka schema registry, which compatibility mode ensures that schemas are compatible both backward and forward?

A Full compatibility

B Forward compatibility

C No compatibility

D Backward compatibility

💡 Hint

Think about the mode that combines both backward and forward compatibility.

Full compatibility mode ensures that new schemas can read data produced by old schemas (backward) and old schemas can read data produced by new schemas (forward).

❓ Predict Output

advanced

What error occurs if a required field is removed in a new schema version?

A Kafka topic uses Avro schemas. The original schema has fields 'name' and 'age' (both required). The new schema removes the 'age' field. What happens when the new schema is registered with full compatibility enabled?

Kafka

Original schema: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}
New schema: {"type":"record","name":"User","fields":[{"name":"name","type":"string"}]}

A Schema registry accepts the new schema but warns about data loss.

B Schema registry accepts the new schema without errors.

C Schema registry rejects the new schema due to incompatibility.

D Schema registry accepts the new schema only if 'age' has a default.

💡 Hint

Removing required fields breaks forward compatibility.

Removing a required field breaks forward compatibility because old consumers expect the 'age' field but new data produced with the new schema won't include it (and old schema has no default for it). Full compatibility requires backward and forward compatibility, so the registry rejects the new schema.

🔧 Debug

expert

Why does this schema registration fail under backward compatibility?

Given the original schema and a new schema, the new schema registration fails under backward compatibility. Identify the cause.

Kafka

Original schema: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":["null","int"],"default":null}]}
New schema: {"type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"age","type":"int"}]}

A The new schema adds a new required field without a default.

B The new schema removes the default for 'age', breaking backward compatibility.

C The new schema changes the type of 'name' field.

D The new schema adds an optional field with a default.

💡 Hint

Check changes in field defaults and types.

Backward compatibility requires that new schemas can read old data. Removing the default from a nullable field makes the new schema incompatible with old data that may have nulls.

Practice

1. What does backward compatibility mean in Kafka schema evolution?

easy

A. New consumers can read data produced with the old schema.

B. Old consumers can read data produced with the new schema.

C. Both old and new consumers can read all data versions.

D. Schemas cannot be changed once set.

Solution

1. Step 1: Understand backward compatibility meaning

   Backward compatibility means older consumers can still read data written with a newer schema.

2. Step 2: Match definition to options

   Old consumers can read data produced with the new schema. states old consumers can read new data, which matches backward compatibility.

3. Final Answer:

   Old consumers can read data produced with the new schema. -> Option B

4. Quick Check:

   Backward compatibility = old consumers read new data [OK]

Hint: Backward means old consumers read new data [OK]

Common Mistakes:

• Confusing backward with forward compatibility
• Thinking backward means new consumers read old data
• Assuming schemas cannot change

2. Which of the following is the correct way to declare a new optional field with a default value in an Avro schema for forward compatibility?

easy

A. {"name": "age", "type": "string", "default": "unknown"}

B. {"name": "age", "type": "int", "default": null}

C. {"name": "age", "type": "int", "default": 0}

D. {"name": "age", "type": "int"}

Solution

1. Step 1: Identify forward compatibility rule

   Forward compatibility requires new fields to have default values so new consumers can read data produced with the old schema.

2. Step 2: Check options for default value and type

   {"name": "age", "type": "int", "default": 0} declares an int field with default 0, which is correct for forward compatibility.

3. Final Answer:

   {"name": "age", "type": "int", "default": 0} -> Option C

4. Quick Check:

   New optional field with default = {"name": "age", "type": "int", "default": 0} [OK]

Hint: New fields need defaults for forward compatibility [OK]

Common Mistakes:

• Missing default value for new fields
• Using null default without union type
• Setting wrong type for default

3. Given this Avro schema evolution scenario:

Old schema: {"name": "id", "type": "int"}
New schema: {"name": "id", "type": "int"}, {"name": "name", "type": "string", "default": "unknown"}

What compatibility type does this change support?

medium

A. Backward compatible only

B. Forward compatible only

C. Not compatible

D. Full compatible

Solution

1. Step 1: Analyze schema change

   A new field "name" with a default value "unknown" is added to the schema.

2. Step 2: Determine compatibility

   Adding a field with a default supports both backward and forward compatibility, so it is full compatible.

3. Final Answer:

   Full compatible -> Option D

4. Quick Check:

   Adding field with default = full compatibility [OK]

Hint: Adding defaulted fields means full compatibility [OK]

Common Mistakes:

• Thinking adding fields breaks backward compatibility
• Ignoring default values
• Confusing forward and backward compatibility

4. You have this Avro schema change:

Old schema: {"name": "email", "type": "string"}
New schema: {"name": "email", "type": ["null", "string"], "default": null}

What is the main issue preventing backward compatibility?

medium

A. Changing field type from string to union breaks old consumers.

B. Adding default null makes schema incompatible.

C. Removing the field causes backward incompatibility.

D. No issue; schema is backward compatible.

Solution

1. Step 1: Identify schema change impact

   The field type changed from string to a union of null and string, which is a type change.

2. Step 2: Understand backward compatibility rules

   Changing a field's type breaks backward compatibility because old consumers expect a string, not a union.

3. Final Answer:

   Changing field type from string to union breaks old consumers. -> Option A

4. Quick Check:

   Type change breaks backward compatibility [OK]

Hint: Changing field type breaks backward compatibility [OK]

Common Mistakes:

• Assuming default null causes incompatibility
• Thinking adding union is always safe
• Ignoring type changes impact

5. You want to evolve a Kafka Avro schema to add a new field status that must be backward and forward compatible. Which approach is best?

hard

A. Add status as an optional string field with default value "active".

B. Add status as a required string field without default.

C. Remove an old optional field and add status as required.

D. Change an existing field type to string and add status without default.

Solution

1. Step 1: Understand compatibility requirements

   Backward and forward compatibility require new fields to have default values and be optional.

2. Step 2: Evaluate options

   Add status as an optional string field with default value "active". adds status as optional with default "active", satisfying both compatibilities.

3. Final Answer:

   Add status as an optional string field with default value "active". -> Option A

4. Quick Check:

   Optional field with default ensures full compatibility [OK]

Hint: New fields need defaults for full compatibility [OK]

Common Mistakes:

• Adding required fields without defaults
• Removing fields breaks compatibility
• Changing existing field types