Event schema design in Microservices - System Design Exercise

Design: Event Schema Design for Microservices

Design focuses on event schema structure, versioning, metadata, and validation. It excludes the underlying messaging infrastructure and microservice business logic.

Functional Requirements

FR1: Define a clear and consistent event schema for communication between microservices

FR2: Support multiple event types with versioning to allow backward compatibility

FR3: Ensure events carry enough metadata for tracing and debugging

FR4: Allow schema evolution without breaking existing consumers

FR5: Support validation of event data before publishing

FR6: Enable efficient serialization and deserialization for low latency

Non-Functional Requirements

NFR1: Handle up to 10,000 events per second

NFR2: Event delivery latency should be under 100ms p99

NFR3: Schema changes must not cause downtime or data loss

NFR4: Support 99.9% availability for event publishing and consumption

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Event schema registry or repository

Schema versioning mechanism

Event metadata structure

Validation service or library

Serialization and deserialization tools

Design Patterns

Schema versioning with backward and forward compatibility

Event enrichment with metadata

Schema registry pattern

Contract testing between producers and consumers

Use of self-describing events

Reference Architecture

 +-------------------+       +-------------------+       +-------------------+
 |  Event Producer   | ----> |  Schema Registry  | ----> |  Event Consumer   |
 +-------------------+       +-------------------+       +-------------------+
          |                          |                          |
          |-- Validate event schema -|                          |
          |                          |                          |
          |-- Serialize event ------>|                          |
          |                          |                          |
          |                          |-- Provide schema -------->|
          |                          |                          |
          |                          |                          |-- Deserialize event
          |                          |                          |
          |                          |                          |-- Validate event schema

Components

Event Producer

Any microservice framework

Generates and publishes events following the defined schema

Schema Registry

Confluent Schema Registry or custom service

Stores event schemas with versioning and provides schema validation

Event Consumer

Any microservice framework

Consumes events, validates against schema, and processes them

Request Flow

1. Producer requests the latest schema version from Schema Registry.

2. Producer validates event data against the schema.

3. Producer serializes the event with schema metadata.

4. Producer publishes the event to the messaging system.

5. Consumer receives the event and extracts schema metadata.

6. Consumer fetches the schema from Schema Registry if not cached.

7. Consumer deserializes and validates the event data.

8. Consumer processes the event.

Database Schema

Entities: - Schema: id (PK), event_type, version, schema_definition (JSON or Avro/Protobuf), created_at - Event Metadata: event_id (PK), event_type, schema_version, timestamp, producer_id, correlation_id Relationships: - Each event references one schema version - Schema versions are linked by event_type with incremental version numbers

Scaling Discussion

Bottlenecks

Schema Registry becomes a single point of failure or bottleneck under high load

Event validation adds latency to event publishing and consumption

Schema evolution causing compatibility issues

Large event payloads increasing network and processing overhead

Solutions

Deploy Schema Registry in a highly available cluster with caching at producers and consumers

Use efficient serialization formats like Avro or Protobuf to reduce payload size and speed validation

Implement strict versioning policies and backward compatibility checks

Use event compression and limit event size by splitting large events

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing schema and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Importance of schema versioning and backward compatibility

Role of schema registry in managing event contracts

How metadata helps in tracing and debugging

Trade-offs between schema strictness and flexibility

Scaling strategies for schema registry and validation

Practice

1. What is the main purpose of an event schema in microservices?

easy

A. To define the structure and content of messages exchanged between services

B. To store user data in a database

C. To create user interfaces for microservices

D. To manage network connections between services

Solution

1. Step 1: Understand event schema role

   An event schema defines how messages look when services talk to each other.

2. Step 2: Identify correct purpose

   It ensures all services understand the message format and data.

3. Final Answer:

   To define the structure and content of messages exchanged between services -> Option A

4. Quick Check:

   Event schema = message format [OK]

Hint: Event schema = message format for services [OK]

Common Mistakes:

• Confusing event schema with database storage
• Thinking event schema manages UI or network
• Assuming event schema is about service deployment

2. Which of the following is a correct JSON snippet for an event schema with type and timestamp?

easy

A. {eventType: "OrderCreated", "timestamp": "2024-06-01T12:00:00Z"}

B. {"eventType": OrderCreated, "timestamp": 2024-06-01T12:00:00Z}

C. {"eventType": "OrderCreated", timestamp: "2024-06-01T12:00:00Z"}

D. {"eventType": "OrderCreated", "timestamp": "2024-06-01T12:00:00Z"}

Solution

1. Step 1: Check JSON syntax rules

   Keys and string values must be in double quotes; commas separate pairs.

2. Step 2: Validate each option

   {"eventType": "OrderCreated", "timestamp": "2024-06-01T12:00:00Z"} uses correct quotes and format; others miss quotes or have invalid syntax.

3. Final Answer:

   {"eventType": "OrderCreated", "timestamp": "2024-06-01T12:00:00Z"} -> Option D

4. Quick Check:

   Valid JSON = {"eventType": "OrderCreated", "timestamp": "2024-06-01T12:00:00Z"} [OK]

Hint: JSON keys and strings need double quotes [OK]

Common Mistakes:

• Missing quotes around keys or string values
• Using unquoted date/time strings
• Omitting commas between pairs

3. Given this event schema snippet:

{"eventType": "UserSignedUp", "timestamp": "2024-06-01T10:00:00Z", "data": {"userId": 123, "email": "[e-mail]"}}

What will be the value of data.email in the event?

medium

A. 123

B. "UserSignedUp"

C. "[e-mail]"

D. 2024-06-01T10:00:00Z

Solution

1. Step 1: Locate the data field in the event

   The event has a nested object under "data" with keys "userId" and "email".

2. Step 2: Identify the value of data.email

   The value for "email" is "[e-mail]" as a string.

3. Final Answer:

   "[e-mail]" -> Option C

4. Quick Check:

   data.email = "[e-mail]" [OK]

Hint: Look inside data object for email key [OK]

Common Mistakes:

• Confusing userId with email
• Picking eventType or timestamp instead
• Ignoring nested structure

4. Identify the error in this event schema:

{"eventType": "PaymentProcessed", "timestamp": "2024-06-01T15:00:00Z", "data": {"amount": 100, "currency": USD}}

medium

A. Missing comma after amount key

B. Missing quotes around the currency value USD

C. timestamp format is incorrect

D. eventType should be lowercase

Solution

1. Step 1: Check JSON value types

   String values must be in double quotes; USD is unquoted here.

2. Step 2: Verify other parts

   Comma after amount is present, timestamp format is ISO standard, eventType case is allowed.

3. Final Answer:

   Missing quotes around the currency value USD -> Option B

4. Quick Check:

   Strings need quotes [OK]

Hint: String values must have quotes in JSON [OK]

Common Mistakes:

• Ignoring missing quotes on string values
• Thinking timestamp format is wrong
• Assuming key case matters in JSON

5. You want to design an event schema for a microservice that sends user profile updates. Which design choice improves schema flexibility for future changes?

hard

A. Include a 'metadata' field to hold optional extra info

B. Fix the schema to only allow 'name' and 'email' fields

C. Use a flat schema without nested objects

D. Exclude timestamps to reduce message size

Solution

1. Step 1: Understand schema flexibility needs

   Flexible schemas allow adding new info without breaking existing services.

2. Step 2: Evaluate options for flexibility

   Adding a 'metadata' field lets you add optional data later safely.

3. Final Answer:

   Include a 'metadata' field to hold optional extra info -> Option A

4. Quick Check:

   Optional metadata = flexible schema [OK]

Hint: Add metadata field for optional future data [OK]

Common Mistakes:

• Fixing schema too rigidly limits future changes
• Removing timestamps loses event timing info
• Avoiding nested objects reduces clarity