Eventual consistency in Microservices - System Design Exercise

Design: Distributed Order Management System

Design focuses on how to achieve eventual consistency across microservices for order, inventory, and payment. Out of scope are UI design and detailed payment gateway integration.

Functional Requirements

FR1: Allow users to place orders that update inventory and payment services

FR2: Ensure data across inventory, payment, and order services eventually become consistent

FR3: Support high availability and partition tolerance

FR4: Handle updates asynchronously to avoid blocking user requests

FR5: Provide a way to detect and resolve conflicts if data diverges

Non-Functional Requirements

NFR1: System must handle 10,000 concurrent orders per minute

NFR2: API response latency under 300ms for order placement

NFR3: Availability target of 99.9% uptime

NFR4: Data consistency can be eventual, not immediate

NFR5: Services deployed independently with separate databases

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message broker or event bus for asynchronous communication

Event store or log for tracking changes

Microservices with their own databases

Conflict detection and resolution logic

Monitoring and alerting for data divergence

Design Patterns

Event-driven architecture

Publish-subscribe messaging

Saga pattern for distributed transactions

CQRS (Command Query Responsibility Segregation)

Idempotent message processing

Reference Architecture

          +----------------+          +----------------+
          |  Order Service |          | Inventory Svc  |
          +-------+--------+          +--------+-------+
                  |                            |
                  | 1. Place Order             | 2. Publish OrderCreated event
                  |--------------------------->|
                  |                            |
                  |                            | 3. Update inventory asynchronously
                  |                            |    and publish InventoryUpdated event
                  |                            |--------------------------->
          +-------v--------+          +--------v-------+
          | Payment Service|          | Event Broker   |
          +-------+--------+          +----------------+
                  |                            ^
                  | 4. Listen InventoryUpdated | 
                  |    event and process       |
                  |                            |
                  | 5. Publish PaymentProcessed|
                  |    event                   |
                  |--------------------------->
                  |                            |
          +-------v--------+                   |
          | Order Service  |<------------------+
          +----------------+

Components

Order Service

REST API, Node.js or Java Spring Boot

Handles order placement and updates order status based on events

Inventory Service

Microservice with own database (e.g., PostgreSQL)

Manages product stock levels and publishes inventory update events

Payment Service

Microservice with own database

Processes payments and publishes payment status events

Event Broker

Apache Kafka or RabbitMQ

Asynchronous message passing to enable eventual consistency

Request Flow

1. User sends order request to Order Service.

2. Order Service validates and saves order with status 'Pending'.

3. Order Service publishes 'OrderCreated' event to Event Broker.

4. Inventory Service consumes 'OrderCreated' event, updates stock, and publishes 'InventoryUpdated' event.

5. Payment Service consumes 'InventoryUpdated' event, processes payment, and publishes 'PaymentProcessed' event.

6. Order Service consumes 'PaymentProcessed' event and updates order status to 'Completed' or 'Failed'.

7. All services process events asynchronously, allowing data to become consistent over time.

Database Schema

Entities: - Order: order_id (PK), user_id, status, total_amount, created_at - Inventory: product_id (PK), stock_quantity - Payment: payment_id (PK), order_id (FK), status, amount, processed_at Relationships: - Order to Payment: 1:1 (order_id foreign key in Payment) - Inventory independent per product Each microservice owns its database to ensure loose coupling.

Scaling Discussion

Bottlenecks

Event Broker throughput limits with high message volume

Database write contention under heavy load

Delayed event processing causing stale data

Handling duplicate or out-of-order events

Conflict resolution complexity when data diverges

Solutions

Partition event topics and scale brokers horizontally

Use database sharding or replicas to distribute load

Implement backpressure and monitoring to detect delays

Design idempotent event handlers and use event versioning

Apply conflict resolution strategies like last-write-wins or manual reconciliation

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why eventual consistency fits the use case and trade-offs involved

Describe asynchronous communication and event-driven design

Highlight how each microservice owns its data and communicates via events

Discuss how to handle failures and conflicts gracefully

Show awareness of scaling challenges and mitigation strategies

Practice

1. What does eventual consistency mean in microservices?

easy

A. Data updates will be visible to all parts of the system after some delay

B. Data is always instantly consistent across all services

C. Data is never synchronized between services

D. Data updates happen only during system maintenance

Solution

1. Step 1: Understand the meaning of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate over time.

2. Step 2: Compare options with the definition

   Only Data updates will be visible to all parts of the system after some delay correctly states that data updates become visible after some delay, matching eventual consistency.

3. Final Answer:

   Data updates will be visible to all parts of the system after some delay -> Option A

4. Quick Check:

   Eventual consistency = delayed data visibility [OK]

Hint: Eventual means "eventually", not instantly [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming updates only during maintenance

2. Which of the following is a correct way to implement eventual consistency in microservices?

easy

A. Use synchronous HTTP calls between services for every update

B. Use asynchronous event messaging to propagate changes

C. Block all reads until all writes complete

D. Disable communication between services

Solution

1. Step 1: Identify communication style for eventual consistency

   Eventual consistency relies on asynchronous communication to allow updates to propagate over time.

2. Step 2: Evaluate options

   Only Use asynchronous event messaging to propagate changes uses asynchronous event messaging, which fits eventual consistency. Others use synchronous or block reads, which do not.

3. Final Answer:

   Use asynchronous event messaging to propagate changes -> Option B

4. Quick Check:

   Asynchronous messaging = eventual consistency [OK]

Hint: Eventual consistency needs async events, not sync calls [OK]

Common Mistakes:

• Choosing synchronous calls which block updates
• Blocking reads causing poor availability
• Ignoring communication between services

3. Consider a microservice system where Service A updates data and publishes an event. Service B listens and updates its copy asynchronously. What is the expected state of Service B immediately after Service A's update?

medium

A. Service B has stale data until it processes the event

B. Service B rejects the update

C. Service B has the updated data instantly

D. Service B crashes due to inconsistency

Solution

1. Step 1: Understand asynchronous event propagation

   Service B updates data only after receiving and processing the event from Service A, which takes time.

2. Step 2: Determine Service B's state immediately after Service A's update

   Since event processing is asynchronous, Service B still holds old data until it processes the event.

3. Final Answer:

   Service B has stale data until it processes the event -> Option A

4. Quick Check:

   Async update means stale data initially [OK]

Hint: Async updates cause temporary stale data [OK]

Common Mistakes:

• Assuming instant data sync
• Thinking services reject updates
• Believing system crashes on inconsistency

4. A microservice system uses event-driven updates but sometimes Service B never receives events from Service A, causing stale data. What is the best fix?

medium

A. Switch to synchronous calls only

B. Ignore the problem as eventual consistency tolerates it

C. Implement event retry and dead-letter queues

D. Stop Service B from reading data

Solution

1. Step 1: Identify problem cause

   Missing events cause stale data because messages are lost or not delivered.

2. Step 2: Choose solution to ensure event delivery

   Implementing retries and dead-letter queues helps guarantee events reach Service B or are logged for manual handling.

3. Final Answer:

   Implement event retry and dead-letter queues -> Option C

4. Quick Check:

   Retries fix lost events = better consistency [OK]

Hint: Use retries and dead-letter queues for reliable events [OK]

Common Mistakes:

• Switching to sync calls losing scalability
• Ignoring lost events causing stale data
• Disabling reads instead of fixing events

5. You design a microservices system with eventual consistency. Service A updates inventory and publishes events. Service B updates order status based on inventory events. How do you ensure order status eventually matches inventory without blocking user requests?

hard

A. Store all data in a single database to avoid events

B. Make Service B synchronously call Service A for every order update

C. Block user requests until all services are consistent

D. Use asynchronous event processing with idempotent handlers and retries

Solution

1. Step 1: Understand requirements for eventual consistency and availability

   The system must update order status eventually without blocking user requests, so async processing is needed.

2. Step 2: Choose design that supports async updates safely

   Using asynchronous event processing with idempotent handlers and retries ensures updates happen reliably and without blocking.

3. Step 3: Evaluate other options

   Synchronous calls or blocking requests reduce availability; single database removes microservices benefits.

4. Final Answer:

   Use asynchronous event processing with idempotent handlers and retries -> Option D

5. Quick Check:

   Async + idempotent + retries = safe eventual consistency [OK]

Hint: Async with retries and idempotency ensures safe updates [OK]

Common Mistakes:

• Blocking user requests hurting availability
• Using sync calls causing tight coupling
• Ignoring idempotency causing duplicate updates