Eventual consistency in Microservices - System Design Exercise

Design: Distributed Order Management System

Design focuses on how to achieve eventual consistency across microservices for order, inventory, and payment. Out of scope are UI design and detailed payment gateway integration.

Functional Requirements

FR1: Allow users to place orders that update inventory and payment services

FR2: Ensure data across inventory, payment, and order services eventually become consistent

FR3: Support high availability and partition tolerance

FR4: Handle updates asynchronously to avoid blocking user requests

FR5: Provide a way to detect and resolve conflicts if data diverges

Non-Functional Requirements

NFR1: System must handle 10,000 concurrent orders per minute

NFR2: API response latency under 300ms for order placement

NFR3: Availability target of 99.9% uptime

NFR4: Data consistency can be eventual, not immediate

NFR5: Services deployed independently with separate databases

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message broker or event bus for asynchronous communication

Event store or log for tracking changes

Microservices with their own databases

Conflict detection and resolution logic

Monitoring and alerting for data divergence

Design Patterns

Event-driven architecture

Publish-subscribe messaging

Saga pattern for distributed transactions

CQRS (Command Query Responsibility Segregation)

Idempotent message processing

Reference Architecture

          +----------------+          +----------------+
          |  Order Service |          | Inventory Svc  |
          +-------+--------+          +--------+-------+
                  |                            |
                  | 1. Place Order             | 2. Publish OrderCreated event
                  |--------------------------->|
                  |                            |
                  |                            | 3. Update inventory asynchronously
                  |                            |    and publish InventoryUpdated event
                  |                            |--------------------------->
          +-------v--------+          +--------v-------+
          | Payment Service|          | Event Broker   |
          +-------+--------+          +----------------+
                  |                            ^
                  | 4. Listen InventoryUpdated | 
                  |    event and process       |
                  |                            |
                  | 5. Publish PaymentProcessed|
                  |    event                   |
                  |--------------------------->
                  |                            |
          +-------v--------+                   |
          | Order Service  |<------------------+
          +----------------+

Components

Order Service

REST API, Node.js or Java Spring Boot

Handles order placement and updates order status based on events

Inventory Service

Microservice with own database (e.g., PostgreSQL)

Manages product stock levels and publishes inventory update events

Payment Service

Microservice with own database

Processes payments and publishes payment status events

Event Broker

Apache Kafka or RabbitMQ

Asynchronous message passing to enable eventual consistency

Request Flow

1. User sends order request to Order Service.

2. Order Service validates and saves order with status 'Pending'.

3. Order Service publishes 'OrderCreated' event to Event Broker.

4. Inventory Service consumes 'OrderCreated' event, updates stock, and publishes 'InventoryUpdated' event.

5. Payment Service consumes 'InventoryUpdated' event, processes payment, and publishes 'PaymentProcessed' event.

6. Order Service consumes 'PaymentProcessed' event and updates order status to 'Completed' or 'Failed'.

7. All services process events asynchronously, allowing data to become consistent over time.

Database Schema

Entities: - Order: order_id (PK), user_id, status, total_amount, created_at - Inventory: product_id (PK), stock_quantity - Payment: payment_id (PK), order_id (FK), status, amount, processed_at Relationships: - Order to Payment: 1:1 (order_id foreign key in Payment) - Inventory independent per product Each microservice owns its database to ensure loose coupling.

Scaling Discussion

Bottlenecks

Event Broker throughput limits with high message volume

Database write contention under heavy load

Delayed event processing causing stale data

Handling duplicate or out-of-order events

Conflict resolution complexity when data diverges

Solutions

Partition event topics and scale brokers horizontally

Use database sharding or replicas to distribute load

Implement backpressure and monitoring to detect delays

Design idempotent event handlers and use event versioning

Apply conflict resolution strategies like last-write-wins or manual reconciliation

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why eventual consistency fits the use case and trade-offs involved

Describe asynchronous communication and event-driven design

Highlight how each microservice owns its data and communicates via events

Discuss how to handle failures and conflicts gracefully

Show awareness of scaling challenges and mitigation strategies