Event schema design in Microservices - System Design Exercise

Design: Event Schema Design for Microservices

Design focuses on event schema structure, versioning, metadata, and validation. It excludes the underlying messaging infrastructure and microservice business logic.

Functional Requirements

FR1: Define a clear and consistent event schema for communication between microservices

FR2: Support multiple event types with versioning to allow backward compatibility

FR3: Ensure events carry enough metadata for tracing and debugging

FR4: Allow schema evolution without breaking existing consumers

FR5: Support validation of event data before publishing

FR6: Enable efficient serialization and deserialization for low latency

Non-Functional Requirements

NFR1: Handle up to 10,000 events per second

NFR2: Event delivery latency should be under 100ms p99

NFR3: Schema changes must not cause downtime or data loss

NFR4: Support 99.9% availability for event publishing and consumption

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Event schema registry or repository

Schema versioning mechanism

Event metadata structure

Validation service or library

Serialization and deserialization tools

Design Patterns

Schema versioning with backward and forward compatibility

Event enrichment with metadata

Schema registry pattern

Contract testing between producers and consumers

Use of self-describing events

Reference Architecture

 +-------------------+       +-------------------+       +-------------------+
 |  Event Producer   | ----> |  Schema Registry  | ----> |  Event Consumer   |
 +-------------------+       +-------------------+       +-------------------+
          |                          |                          |
          |-- Validate event schema -|                          |
          |                          |                          |
          |-- Serialize event ------>|                          |
          |                          |                          |
          |                          |-- Provide schema -------->|
          |                          |                          |
          |                          |                          |-- Deserialize event
          |                          |                          |
          |                          |                          |-- Validate event schema

Components

Event Producer

Any microservice framework

Generates and publishes events following the defined schema

Schema Registry

Confluent Schema Registry or custom service

Stores event schemas with versioning and provides schema validation

Event Consumer

Any microservice framework

Consumes events, validates against schema, and processes them

Request Flow

1. Producer requests the latest schema version from Schema Registry.

2. Producer validates event data against the schema.

3. Producer serializes the event with schema metadata.

4. Producer publishes the event to the messaging system.

5. Consumer receives the event and extracts schema metadata.

6. Consumer fetches the schema from Schema Registry if not cached.

7. Consumer deserializes and validates the event data.

8. Consumer processes the event.

Database Schema

Entities: - Schema: id (PK), event_type, version, schema_definition (JSON or Avro/Protobuf), created_at - Event Metadata: event_id (PK), event_type, schema_version, timestamp, producer_id, correlation_id Relationships: - Each event references one schema version - Schema versions are linked by event_type with incremental version numbers

Scaling Discussion

Bottlenecks

Schema Registry becomes a single point of failure or bottleneck under high load

Event validation adds latency to event publishing and consumption

Schema evolution causing compatibility issues

Large event payloads increasing network and processing overhead

Solutions

Deploy Schema Registry in a highly available cluster with caching at producers and consumers

Use efficient serialization formats like Avro or Protobuf to reduce payload size and speed validation

Implement strict versioning policies and backward compatibility checks

Use event compression and limit event size by splitting large events

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing schema and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Importance of schema versioning and backward compatibility

Role of schema registry in managing event contracts

How metadata helps in tracing and debugging

Trade-offs between schema strictness and flexibility

Scaling strategies for schema registry and validation