Event store concept in Microservices - System Design Exercise

Design: Event Store System

Design the core event store service and its API for microservices. Out of scope: event consumer implementations, UI, and event processing logic.

Functional Requirements

FR1: Store all changes to application state as a sequence of events.

FR2: Support appending new events in order with strong consistency.

FR3: Allow querying events by aggregate or event type.

FR4: Support replaying events to rebuild current state.

FR5: Provide an API for microservices to publish and consume events.

FR6: Ensure durability and fault tolerance of stored events.

Non-Functional Requirements

NFR1: Handle up to 10,000 events per second.

NFR2: API response latency under 200ms for event appends and queries.

NFR3: 99.9% availability (less than 8.77 hours downtime per year).

NFR4: Events are immutable and never deleted.

NFR5: Support eventual consistency for event consumers.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

API gateway or event ingestion service

Event storage database (e.g., append-only log store)

Event indexing and query service

Event replay mechanism

Message broker or pub/sub system for event distribution

Monitoring and alerting tools

Design Patterns

Event Sourcing pattern

CQRS (Command Query Responsibility Segregation)

Append-only log storage

Publish-Subscribe messaging

Snapshotting for state rebuild optimization

Reference Architecture

                +---------------------+
                |  Microservice Client |
                +----------+----------+
                           |
                           | REST/gRPC API calls
                           v
                +----------+----------+
                |   Event Store API   |
                +----------+----------+
                           |
           +---------------+---------------+
           |                               |
           v                               v
+-------------------+           +---------------------+
| Event Storage DB  |           |  Message Broker      |
| (Append-only log) |           | (Kafka/RabbitMQ)    |
+-------------------+           +---------------------+
           |                               |
           |                               v
           |                    +---------------------+
           |                    | Event Consumers      |
           |                    +---------------------+
           |
           v
+-------------------+
| Event Query Index  |
+-------------------+

Components

Event Store API

REST/gRPC service

Receives event append requests and query requests from microservices.

Event Storage DB

Append-only log database (e.g., Apache Cassandra, EventStoreDB)

Stores all events immutably in order for durability and replay.

Message Broker

Kafka or RabbitMQ

Distributes new events to interested consumers asynchronously.

Event Query Index

Search engine or NoSQL DB (e.g., Elasticsearch, DynamoDB)

Provides fast querying of events by aggregate or type.

Event Consumers

Microservices or workers

Subscribe to events for processing and updating read models.

Request Flow

1. 1. Client microservice sends event append request to Event Store API.

2. 2. Event Store API validates and appends event to Event Storage DB in order.

3. 3. Event Store API publishes event to Message Broker for consumers.

4. 4. Event Query Index updates asynchronously for fast event queries.

5. 5. Event consumers subscribe to Message Broker to receive new events.

6. 6. Consumers process events and update their own state or read models.

7. 7. Clients query Event Store API to retrieve events by aggregate or type.

Database Schema

Entities: - Event: {event_id (PK), aggregate_id, aggregate_type, event_type, event_data (JSON), timestamp, version} Relationships: - Events are linked by aggregate_id and version to maintain order per aggregate. - No deletion or update of events; append-only storage. - Index on aggregate_id and event_type for efficient queries.

Scaling Discussion

Bottlenecks

Event Storage DB write throughput limit at high event rates.

Message Broker throughput and retention limits.

Query Index lagging behind event ingestion causing stale reads.

Event Store API becoming a single point of failure under load.

Solutions

Partition event storage by aggregate_id or time to scale writes horizontally.

Use a distributed, scalable message broker like Kafka with topic partitioning.

Implement asynchronous indexing with backpressure and monitoring to keep index updated.

Deploy Event Store API as a stateless service behind a load balancer with autoscaling.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain event sourcing benefits: auditability, state rebuild, decoupling.

Discuss immutability and append-only storage importance.

Highlight how message brokers enable asynchronous event distribution.

Mention indexing for efficient queries and snapshotting for optimization.

Address scaling challenges and solutions realistically.