Why schema management prevents data issues in Kafka - See It in Action

Why schema management prevents data issues

📋 What You'll Learn

💡 Why This Matters

🌍 Real World

Many companies use Kafka to move data between services. Schema management keeps data consistent and prevents bugs when formats change.

💼 Career

Understanding schema management is important for roles like data engineer, backend developer, and system architect working with streaming data.

1

Create a message schema

Create a variable called user_schema that holds the Avro schema string for a user with fields name (string) and age (int). Use the exact schema format shown.

Kafka

# Your code here

Need a hint?

The schema is a JSON string that defines the data structure. Use triple quotes for multi-line string.

2

Configure the Kafka producer with schema

Create a variable called producer_config as a dictionary with keys bootstrap.servers set to localhost:9092 and schema.registry.url set to http://localhost:8081.

Kafka

user_schema = '''{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "name", "type": "string"},
    {"name": "age", "type": "int"}
  ]
}'''

# Your code here

Need a hint?

This config tells Kafka where the server and schema registry are.

3

Serialize and send a message using the schema

Use the producer_config and user_schema to create a Kafka Avro producer called producer. Then send a message with name as "Alice" and age as 30 to the topic users.

Kafka

user_schema = '''{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "name", "type": "string"},
    {"name": "age", "type": "int"}
  ]
}'''

producer_config = {
    "bootstrap.servers": "localhost:9092",
    "schema.registry.url": "http://localhost:8081"
}

# Your code here

Need a hint?

Use AvroProducer with default_value_schema loaded from user_schema. Then call produce with the message and flush to send.

4

Consume and print the message using the schema

Create a Kafka Avro consumer called consumer with the same config as producer_config. Subscribe to the topic users. Poll one message, then print the value of the message.

Kafka

from confluent_kafka.avro import AvroProducer, AvroConsumer
import json

user_schema = '''{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "name", "type": "string"},
    {"name": "age", "type": "int"}
  ]
}'''

producer_config = {
    "bootstrap.servers": "localhost:9092",
    "schema.registry.url": "http://localhost:8081"
}

producer = AvroProducer(producer_config, default_value_schema=json.loads(user_schema))

producer.produce(topic="users", value={"name": "Alice", "age": 30})
producer.flush()

# Your code here

Need a hint?

Use AvroConsumer with the same config. Subscribe to users. Poll a message and print its value.