Event sourcing pattern in Microservices - System Design Exercise

Design: Event Sourcing System

Design the event sourcing pattern implementation within a microservices architecture including event storage, event publishing, state rebuilding, and querying. Out of scope: UI design, specific business logic details.

Functional Requirements

FR1: Store all changes to application state as a sequence of events.

FR2: Allow rebuilding current state by replaying events.

FR3: Support querying current state efficiently.

FR4: Ensure events are immutable and append-only.

FR5: Support multiple microservices consuming events asynchronously.

FR6: Provide audit trail for all changes.

FR7: Handle concurrent updates safely.

Non-Functional Requirements

NFR1: System must handle 10,000 events per second.

NFR2: Event replay latency should be under 5 seconds for rebuilding state.

NFR3: Availability target of 99.9% uptime.

NFR4: Event storage must be durable and scalable.

NFR5: APIs should respond within 200ms p99 latency.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

❓ Question 7

Key Components

Event Store (append-only log database)

Command API to accept state changes

Event Publisher / Message Broker

Event Consumers / Microservices

Snapshot Store for faster state rebuilding

Read Model / Query Database

Concurrency control mechanisms

Design Patterns

Event Sourcing

CQRS (Command Query Responsibility Segregation)

Publish-Subscribe messaging

Snapshotting

Idempotent event processing

Reference Architecture

          +-------------------+          
          |   Client / UI     |          
          +---------+---------+          
                    |                    
                    v                    
          +---------+---------+          
          |   Command API     |          
          +---------+---------+          
                    |                    
                    v                    
          +---------+---------+          
          |    Event Store    |<---------+  
          +---------+---------+          |  
                    |                    |  
                    v                    |  
          +---------+---------+          |  
          | Event Publisher   |          |  
          +---------+---------+          |  
                    |                    |  
        +-----------+-----------+        |  
        |                       |        |  
+-------v-------+       +-------v-------+|  
| Read Model DB |       | Microservice 1 ||  
+---------------+       +---------------+|  
                                         |  
          +-------------------------------+  
          |       Snapshot Store           
          +-------------------------------+  

Components

Command API

REST/gRPC microservice

Accepts commands from clients to change state, validates and converts them into events.

Event Store

Append-only log database (e.g., Apache Kafka, EventStoreDB)

Stores all events immutably in order for replay and audit.

Event Publisher

Message broker (e.g., Kafka, RabbitMQ)

Publishes events to subscribers asynchronously.

Microservices (Event Consumers)

Microservices subscribing to event streams

Consume events to update their own state or trigger actions.

Read Model Database

NoSQL or relational DB optimized for queries

Stores current state projections built from events for fast querying.

Snapshot Store

Key-value store or database

Stores periodic snapshots of state to speed up rebuilding from events.

Request Flow

1. Client sends a command to Command API to change state.

2. Command API validates command and creates one or more events.

3. Events are appended to the Event Store in order.

4. Event Publisher reads new events from Event Store and publishes them to message broker.

5. Microservices subscribe to event streams and consume events asynchronously.

6. Microservices update their own state or trigger side effects based on events.

7. Read Model Database is updated by event consumers to reflect current state.

8. Snapshot Store periodically saves snapshots of state to optimize rebuilding.

9. When rebuilding state, system loads latest snapshot and replays subsequent events.

Database Schema

Entities: - Event: {event_id (PK), aggregate_id, event_type, event_data (JSON), timestamp, version} - Snapshot: {snapshot_id (PK), aggregate_id, snapshot_data (JSON), last_event_version, timestamp} - Aggregate: {aggregate_id (PK), current_version, last_snapshot_version} Relationships: - One Aggregate has many Events (1:N) - One Aggregate has many Snapshots (1:N) - Events are immutable and append-only - Snapshots reference the last event version included

Scaling Discussion

Bottlenecks

Event Store write throughput limits at very high event rates.

Event replay latency grows with event history size.

Message broker overload with many subscribers or high event volume.

Read Model DB query performance under heavy load.

Snapshot creation impacting system performance.

Solutions

Partition Event Store by aggregate or topic to increase write throughput.

Use snapshotting to reduce event replay time for rebuilding state.

Scale message broker clusters and use consumer groups for load balancing.

Use read model databases optimized for query patterns and scale horizontally.

Schedule snapshot creation during low traffic periods and use incremental snapshots.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 15 minutes designing components and data flow, 10 minutes discussing scaling and trade-offs, 10 minutes for questions and summary.

Explain why event sourcing is useful for auditability and state reconstruction.

Describe how events are stored immutably and published asynchronously.

Discuss the role of snapshots to optimize rebuilding state.

Highlight how CQRS separates command and query responsibilities.

Address concurrency and ordering challenges in event processing.

Mention scaling strategies for event store, message broker, and read models.

Practice

1. What is the main idea behind the Event sourcing pattern in microservices?

easy

A. Store all changes as a sequence of events instead of only the current state

B. Store only the latest snapshot of data for faster access

C. Use events only for communication between services, not for storage

D. Store events only when errors occur in the system

Solution

1. Step 1: Understand event sourcing concept

   Event sourcing means saving every change as an event, not just the final state.

2. Step 2: Compare options with definition

   Only Store all changes as a sequence of events instead of only the current state correctly describes storing all changes as events, others focus on snapshots or partial use.

3. Final Answer:

   Store all changes as a sequence of events instead of only the current state -> Option A

4. Quick Check:

   Event sourcing = storing all changes as events [OK]

Hint: Event sourcing saves every change, not just the final state [OK]

Common Mistakes:

• Confusing event sourcing with snapshot storage
• Thinking events are only for communication
• Believing events are stored only on errors

2. Which of the following is the correct sequence in event sourcing when a user updates their profile?

easy

A. Update state directly, then create an event for the change

B. Send event to other services before updating state

C. Create an event representing the change, then update the state from the event

D. Store the new state without creating any event

Solution

1. Step 1: Recall event sourcing flow

   In event sourcing, changes are first recorded as events, then state is updated from those events.

2. Step 2: Match options to flow

   Create an event representing the change, then update the state from the event correctly shows event creation before state update; others do not follow this order.

3. Final Answer:

   Create an event representing the change, then update the state from the event -> Option C

4. Quick Check:

   Event first, then state update = Create an event representing the change, then update the state from the event [OK]

Hint: Events come before state updates in event sourcing [OK]

Common Mistakes:

• Updating state before creating events
• Skipping event creation
• Sending events before state is updated

3. Given this simplified event sourcing code snippet, what will be the final user balance?

events = [
  {"type": "Deposit", "amount": 100},
  {"type": "Withdraw", "amount": 30},
  {"type": "Deposit", "amount": 50}
]

balance = 0
for event in events:
    if event["type"] == "Deposit":
        balance += event["amount"]
    elif event["type"] == "Withdraw":
        balance -= event["amount"]
print(balance)

medium

A. 80

B. 150

C. 100

D. 120

Solution

1. Step 1: Calculate deposits and withdrawals

   Deposits: 100 + 50 = 150; Withdrawals: 30

2. Step 2: Compute final balance

   Balance = 0 + 150 - 30 = 120

3. Final Answer:

   120 -> Option D

4. Quick Check:

   Deposits minus withdrawals = 120 [OK]

Hint: Add deposits, subtract withdrawals to find balance [OK]

Common Mistakes:

• Adding withdrawals instead of subtracting
• Ignoring last deposit event
• Starting balance not zero

4. In an event sourcing system, which issue is caused by missing an event during state reconstruction?

medium

A. System crashes immediately

B. State becomes inconsistent or incorrect

C. Events are duplicated in the event store

D. Commands fail to generate new events

Solution

1. Step 1: Understand event replay in event sourcing

   State is rebuilt by replaying all events in order.

2. Step 2: Effect of missing an event

   If an event is missing, the rebuilt state will not reflect all changes, causing inconsistency.

3. Final Answer:

   State becomes inconsistent or incorrect -> Option B

4. Quick Check:

   Missing event = incorrect state [OK]

Hint: Missing events cause wrong state, not crashes [OK]

Common Mistakes:

• Assuming system crashes on missing event
• Confusing missing event with duplicate events
• Thinking commands stop working due to missing event

5. You are designing a microservice using event sourcing. To improve performance, you want to avoid replaying all events every time. Which approach is best to achieve this?

hard

A. Use snapshots to save periodic state and replay only recent events

B. Store only the latest event and discard older ones

C. Send all events to other services for processing

D. Update state directly and ignore events after initial load

Solution

1. Step 1: Identify performance issue in event sourcing

   Replaying all events from the start can be slow as event count grows.

2. Step 2: Choose solution to reduce replay time

   Snapshots save the current state at intervals, so only recent events need replaying.

3. Final Answer:

   Use snapshots to save periodic state and replay only recent events -> Option A

4. Quick Check:

   Snapshots improve replay speed [OK]

Hint: Snapshots speed up state rebuild by reducing event replay [OK]

Common Mistakes:

• Discarding old events breaks event sourcing
• Sending all events elsewhere doesn't reduce replay
• Ignoring events after initial load loses audit trail