Data consistency challenges in Microservices - System Design Exercise

Design: Microservices Data Consistency Management

Focus on data consistency challenges in microservices architecture including transaction management, conflict resolution, and consistency models. Out of scope are UI design and specific business logic implementations.

Functional Requirements

FR1: Ensure data consistency across multiple microservices managing related data.

FR2: Support eventual consistency where strict consistency is not feasible.

FR3: Handle distributed transactions or compensate for failures gracefully.

FR4: Provide mechanisms to detect and resolve data conflicts.

FR5: Maintain system availability and responsiveness during data updates.

Non-Functional Requirements

NFR1: System must handle 10,000 concurrent requests updating related data.

NFR2: API response latency p99 should be under 300ms for read operations.

NFR3: Availability target is 99.9% uptime (less than 8.77 hours downtime per year).

NFR4: Microservices are independently deployable and scalable.

NFR5: No single point of failure in data consistency mechanisms.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

API Gateway or Service Mesh for communication

Event Bus or Message Queue for asynchronous updates

Distributed Transaction Coordinator or Saga Orchestrator

Databases per microservice with local transactions

Conflict detection and resolution mechanisms

Design Patterns

Saga pattern for managing distributed transactions

Eventual consistency with event-driven architecture

Two-phase commit (2PC) for strong consistency (with trade-offs)

Idempotency to handle retries safely

CQRS (Command Query Responsibility Segregation) for read/write separation

Reference Architecture

          +-------------------+          
          |   API Gateway     |          
          +---------+---------+          
                    |                    
        +-----------+-----------+        
        |                       |        
+-------v-------+       +-------v-------+
| Order Service |       | Payment Service|
+-------+-------+       +-------+-------+
        |                       |        
        |                       |        
+-------v-------+       +-------v-------+
| Order DB      |       | Payment DB    |
+---------------+       +---------------+
        |                       |        
        +-----------+-----------+        
                    |                    
             +------v-------+            
             | Event Broker |            
             +--------------+            
                    |                    
          +---------+---------+          
          |   Notification     |          
          |     Service        |          
          +-------------------+          

Components

API Gateway

Nginx / Kong / Istio

Entry point for client requests routing to appropriate microservices.

Order Service

Spring Boot / Node.js / Go

Handles order creation and updates with local database transactions.

Payment Service

Spring Boot / Node.js / Go

Processes payments independently with its own database.

Order DB

PostgreSQL / MySQL

Stores order data with ACID transactions.

Payment DB

PostgreSQL / MySQL

Stores payment data with ACID transactions.

Event Broker

Kafka / RabbitMQ

Facilitates asynchronous event-driven communication between services.

Notification Service

Node.js / Python

Sends notifications based on events from other services.

Saga Orchestrator

Custom or framework-based (e.g., Temporal.io)

Coordinates distributed transactions across services using saga pattern.

Request Flow

1. Client sends order creation request to API Gateway.

2. API Gateway routes request to Order Service.

3. Order Service creates order in Order DB within local transaction.

4. Order Service publishes 'OrderCreated' event to Event Broker.

5. Saga Orchestrator listens to 'OrderCreated' event and triggers Payment Service.

6. Payment Service processes payment and updates Payment DB.

7. Payment Service publishes 'PaymentProcessed' event to Event Broker.

8. Saga Orchestrator listens to 'PaymentProcessed' event and confirms order completion.

9. If any step fails, Saga Orchestrator triggers compensating transactions to rollback.

10. Notification Service listens to events and sends updates to users asynchronously.

Database Schema

Entities: - Order: id (PK), user_id, status, total_amount, created_at, updated_at - Payment: id (PK), order_id (FK), payment_status, amount, payment_method, created_at, updated_at Relationships: - One-to-One from Order to Payment via order_id - Each service owns its database to ensure local ACID transactions - Event Broker decouples data synchronization between services

Scaling Discussion

Bottlenecks

Event Broker throughput limits with high event volume.

Saga Orchestrator becoming a single point of failure or bottleneck.

Database write contention under heavy load.

Increased latency due to asynchronous event processing.

Complexity in handling compensating transactions and failure scenarios.

Solutions

Partition Event Broker topics and scale brokers horizontally.

Implement Saga Orchestrator as a distributed, fault-tolerant service.

Use database sharding or read replicas to distribute load.

Optimize event processing with parallel consumers and backpressure handling.

Design idempotent and retryable operations; use monitoring and alerting for failures.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling challenges and trade-offs, and 5 minutes summarizing key points.

Explain the trade-offs between strong and eventual consistency in microservices.

Describe how the saga pattern helps manage distributed transactions.

Discuss the role of event-driven architecture in decoupling services.

Highlight failure handling with compensating transactions.

Address scalability and availability considerations in the design.