Event store concept in Microservices - System Design Exercise

Design: Event Store System

Design the core event store service and its API for microservices. Out of scope: event consumer implementations, UI, and event processing logic.

Functional Requirements

FR1: Store all changes to application state as a sequence of events.

FR2: Support appending new events in order with strong consistency.

FR3: Allow querying events by aggregate or event type.

FR4: Support replaying events to rebuild current state.

FR5: Provide an API for microservices to publish and consume events.

FR6: Ensure durability and fault tolerance of stored events.

Non-Functional Requirements

NFR1: Handle up to 10,000 events per second.

NFR2: API response latency under 200ms for event appends and queries.

NFR3: 99.9% availability (less than 8.77 hours downtime per year).

NFR4: Events are immutable and never deleted.

NFR5: Support eventual consistency for event consumers.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

API gateway or event ingestion service

Event storage database (e.g., append-only log store)

Event indexing and query service

Event replay mechanism

Message broker or pub/sub system for event distribution

Monitoring and alerting tools

Design Patterns

Event Sourcing pattern

CQRS (Command Query Responsibility Segregation)

Append-only log storage

Publish-Subscribe messaging

Snapshotting for state rebuild optimization

Reference Architecture

                +---------------------+
                |  Microservice Client |
                +----------+----------+
                           |
                           | REST/gRPC API calls
                           v
                +----------+----------+
                |   Event Store API   |
                +----------+----------+
                           |
           +---------------+---------------+
           |                               |
           v                               v
+-------------------+           +---------------------+
| Event Storage DB  |           |  Message Broker      |
| (Append-only log) |           | (Kafka/RabbitMQ)    |
+-------------------+           +---------------------+
           |                               |
           |                               v
           |                    +---------------------+
           |                    | Event Consumers      |
           |                    +---------------------+
           |
           v
+-------------------+
| Event Query Index  |
+-------------------+

Components

Event Store API

REST/gRPC service

Receives event append requests and query requests from microservices.

Event Storage DB

Append-only log database (e.g., Apache Cassandra, EventStoreDB)

Stores all events immutably in order for durability and replay.

Message Broker

Kafka or RabbitMQ

Distributes new events to interested consumers asynchronously.

Event Query Index

Search engine or NoSQL DB (e.g., Elasticsearch, DynamoDB)

Provides fast querying of events by aggregate or type.

Event Consumers

Microservices or workers

Subscribe to events for processing and updating read models.

Request Flow

1. 1. Client microservice sends event append request to Event Store API.

2. 2. Event Store API validates and appends event to Event Storage DB in order.

3. 3. Event Store API publishes event to Message Broker for consumers.

4. 4. Event Query Index updates asynchronously for fast event queries.

5. 5. Event consumers subscribe to Message Broker to receive new events.

6. 6. Consumers process events and update their own state or read models.

7. 7. Clients query Event Store API to retrieve events by aggregate or type.

Database Schema

Entities: - Event: {event_id (PK), aggregate_id, aggregate_type, event_type, event_data (JSON), timestamp, version} Relationships: - Events are linked by aggregate_id and version to maintain order per aggregate. - No deletion or update of events; append-only storage. - Index on aggregate_id and event_type for efficient queries.

Scaling Discussion

Bottlenecks

Event Storage DB write throughput limit at high event rates.

Message Broker throughput and retention limits.

Query Index lagging behind event ingestion causing stale reads.

Event Store API becoming a single point of failure under load.

Solutions

Partition event storage by aggregate_id or time to scale writes horizontally.

Use a distributed, scalable message broker like Kafka with topic partitioning.

Implement asynchronous indexing with backpressure and monitoring to keep index updated.

Deploy Event Store API as a stateless service behind a load balancer with autoscaling.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain event sourcing benefits: auditability, state rebuild, decoupling.

Discuss immutability and append-only storage importance.

Highlight how message brokers enable asynchronous event distribution.

Mention indexing for efficient queries and snapshotting for optimization.

Address scaling challenges and solutions realistically.

Practice

1. What is the primary purpose of an event store in a microservices architecture?

easy

A. To save every change as an immutable event in order

B. To store user credentials securely

C. To cache frequently accessed data for faster reads

D. To manage service discovery and load balancing

Solution

1. Step 1: Understand event store role

   An event store records all changes as events, preserving order and immutability.

2. Step 2: Compare options with event store purpose

   Only To save every change as an immutable event in order describes saving changes as immutable events in order, which matches event store's main function.

3. Final Answer:

   To save every change as an immutable event in order -> Option A

4. Quick Check:

   Event store = immutable ordered events [OK]

Hint: Event store saves changes as events, not data or cache [OK]

Common Mistakes:

• Confusing event store with caching layer
• Thinking event store manages security or load balancing
• Assuming event store modifies events after saving

2. Which of the following best describes the structure of data in an event store?

easy

A. A mutable key-value store with random access

B. An append-only log of immutable events

C. A relational database with tables and joins

D. A cache with time-to-live expiration

Solution

1. Step 1: Identify event store data structure

   Event stores keep data as an append-only log where events cannot be changed once stored.

2. Step 2: Match options to event store structure

   An append-only log of immutable events correctly describes an append-only log of immutable events, unlike mutable stores or caches.

3. Final Answer:

   An append-only log of immutable events -> Option B

4. Quick Check:

   Event store = append-only immutable log [OK]

Hint: Event store data is append-only and immutable, not mutable [OK]

Common Mistakes:

• Thinking event store allows event updates
• Confusing event store with relational databases
• Assuming event store is a cache with expiration

3. Given the following sequence of events stored in an event store:

1: UserCreated {userId: 1, name: "Alice"}
2: UserNameUpdated {userId: 1, name: "Alicia"}
3: UserDeleted {userId: 1}

What is the current state of the user with userId=1 after replaying these events?

medium

A. User with name "Alice" and deleted flag true

B. User with name "Alicia" exists

C. User with name "Alice" exists

D. User does not exist

Solution

1. Step 1: Replay events in order

   First event creates user Alice, second updates name to Alicia, third deletes the user.

2. Step 2: Determine final user state

   After deletion event, user no longer exists regardless of previous name changes.

3. Final Answer:

   User does not exist -> Option D

4. Quick Check:

   Last event is deletion, so user is gone [OK]

Hint: Last event determines existence; deletion means no user [OK]

Common Mistakes:

• Ignoring the delete event
• Assuming user name remains after deletion
• Confusing event replay order

4. You notice that your event store is allowing events to be updated after they are stored. What is the main issue with this behavior?

medium

A. It enables faster event replay by skipping old events

B. It improves performance by reducing storage needs

C. It breaks the immutability principle, causing inconsistent system state

D. It allows easier debugging by fixing event data

Solution

1. Step 1: Understand immutability in event stores

   Events must be immutable to ensure reliable replay and audit trails.

2. Step 2: Analyze impact of updating events

   Updating events breaks immutability, leading to inconsistent or incorrect system state.

3. Final Answer:

   It breaks the immutability principle, causing inconsistent system state -> Option C

4. Quick Check:

   Event immutability = consistent state [OK]

Hint: Events must never change after storing [OK]

Common Mistakes:

• Thinking event updates improve debugging
• Assuming updates improve performance
• Believing updates speed up replay

5. In a microservices system using an event store, how can you efficiently rebuild the current state of a service that has millions of events without replaying all events every time?

hard

A. Use snapshots to save intermediate states periodically

B. Delete old events after a certain time to reduce replay

C. Store only the latest event per entity to minimize data

D. Replay events in parallel without ordering

Solution

1. Step 1: Identify replay challenges with many events

   Replaying millions of events is slow and inefficient for rebuilding state.

2. Step 2: Evaluate solutions to speed up rebuilding

   Snapshots save the state at points in time, allowing replay from snapshot forward, reducing events to process.

3. Final Answer:

   Use snapshots to save intermediate states periodically -> Option A

4. Quick Check:

   Snapshots optimize replay by reducing event count [OK]

Hint: Snapshots speed up state rebuild, don't delete events [OK]

Common Mistakes:

• Deleting old events breaks audit and consistency
• Storing only latest event loses history
• Replaying events out of order causes errors