Data consistency challenges in Microservices - System Design Exercise

Design: Microservices Data Consistency Management

Focus on data consistency challenges in microservices architecture including transaction management, conflict resolution, and consistency models. Out of scope are UI design and specific business logic implementations.

Functional Requirements

FR1: Ensure data consistency across multiple microservices managing related data.

FR2: Support eventual consistency where strict consistency is not feasible.

FR3: Handle distributed transactions or compensate for failures gracefully.

FR4: Provide mechanisms to detect and resolve data conflicts.

FR5: Maintain system availability and responsiveness during data updates.

Non-Functional Requirements

NFR1: System must handle 10,000 concurrent requests updating related data.

NFR2: API response latency p99 should be under 300ms for read operations.

NFR3: Availability target is 99.9% uptime (less than 8.77 hours downtime per year).

NFR4: Microservices are independently deployable and scalable.

NFR5: No single point of failure in data consistency mechanisms.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

API Gateway or Service Mesh for communication

Event Bus or Message Queue for asynchronous updates

Distributed Transaction Coordinator or Saga Orchestrator

Databases per microservice with local transactions

Conflict detection and resolution mechanisms

Design Patterns

Saga pattern for managing distributed transactions

Eventual consistency with event-driven architecture

Two-phase commit (2PC) for strong consistency (with trade-offs)

Idempotency to handle retries safely

CQRS (Command Query Responsibility Segregation) for read/write separation

Reference Architecture

          +-------------------+          
          |   API Gateway     |          
          +---------+---------+          
                    |                    
        +-----------+-----------+        
        |                       |        
+-------v-------+       +-------v-------+
| Order Service |       | Payment Service|
+-------+-------+       +-------+-------+
        |                       |        
        |                       |        
+-------v-------+       +-------v-------+
| Order DB      |       | Payment DB    |
+---------------+       +---------------+
        |                       |        
        +-----------+-----------+        
                    |                    
             +------v-------+            
             | Event Broker |            
             +--------------+            
                    |                    
          +---------+---------+          
          |   Notification     |          
          |     Service        |          
          +-------------------+          

Components

API Gateway

Nginx / Kong / Istio

Entry point for client requests routing to appropriate microservices.

Order Service

Spring Boot / Node.js / Go

Handles order creation and updates with local database transactions.

Payment Service

Spring Boot / Node.js / Go

Processes payments independently with its own database.

Order DB

PostgreSQL / MySQL

Stores order data with ACID transactions.

Payment DB

PostgreSQL / MySQL

Stores payment data with ACID transactions.

Event Broker

Kafka / RabbitMQ

Facilitates asynchronous event-driven communication between services.

Notification Service

Node.js / Python

Sends notifications based on events from other services.

Saga Orchestrator

Custom or framework-based (e.g., Temporal.io)

Coordinates distributed transactions across services using saga pattern.

Request Flow

1. Client sends order creation request to API Gateway.

2. API Gateway routes request to Order Service.

3. Order Service creates order in Order DB within local transaction.

4. Order Service publishes 'OrderCreated' event to Event Broker.

5. Saga Orchestrator listens to 'OrderCreated' event and triggers Payment Service.

6. Payment Service processes payment and updates Payment DB.

7. Payment Service publishes 'PaymentProcessed' event to Event Broker.

8. Saga Orchestrator listens to 'PaymentProcessed' event and confirms order completion.

9. If any step fails, Saga Orchestrator triggers compensating transactions to rollback.

10. Notification Service listens to events and sends updates to users asynchronously.

Database Schema

Entities: - Order: id (PK), user_id, status, total_amount, created_at, updated_at - Payment: id (PK), order_id (FK), payment_status, amount, payment_method, created_at, updated_at Relationships: - One-to-One from Order to Payment via order_id - Each service owns its database to ensure local ACID transactions - Event Broker decouples data synchronization between services

Scaling Discussion

Bottlenecks

Event Broker throughput limits with high event volume.

Saga Orchestrator becoming a single point of failure or bottleneck.

Database write contention under heavy load.

Increased latency due to asynchronous event processing.

Complexity in handling compensating transactions and failure scenarios.

Solutions

Partition Event Broker topics and scale brokers horizontally.

Implement Saga Orchestrator as a distributed, fault-tolerant service.

Use database sharding or read replicas to distribute load.

Optimize event processing with parallel consumers and backpressure handling.

Design idempotent and retryable operations; use monitoring and alerting for failures.

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling challenges and trade-offs, and 5 minutes summarizing key points.

Explain the trade-offs between strong and eventual consistency in microservices.

Describe how the saga pattern helps manage distributed transactions.

Discuss the role of event-driven architecture in decoupling services.

Highlight failure handling with compensating transactions.

Address scalability and availability considerations in the design.

Practice

1. What is the main challenge of data consistency in microservices?

easy

A. Ensuring all services see the same data at the same time

B. Writing code in multiple programming languages

C. Deploying services on different servers

D. Using different databases for each service

Solution

1. Step 1: Understand data sharing in microservices

   Microservices often manage their own data, but sometimes share data across services.

2. Step 2: Identify the consistency challenge

   Because data is shared, keeping it the same across services at the same time is difficult.

3. Final Answer:

   Ensuring all services see the same data at the same time -> Option A

4. Quick Check:

   Data consistency = same data view [OK]

Hint: Data consistency means same data visible everywhere [OK]

Common Mistakes:

• Confusing deployment issues with data consistency
• Thinking language differences cause consistency problems
• Assuming different databases alone cause consistency issues

2. Which of the following is a common technique to handle temporary data inconsistency in microservices?

easy

A. Using synchronous database locks across services

B. Disabling network retries to avoid duplicate messages

C. Sharing a single database instance for all services

D. Implementing event-driven communication with retries

Solution

1. Step 1: Review methods to handle inconsistency

   Temporary inconsistencies happen due to delays or failures in communication between services.

2. Step 2: Identify best practice

   Event-driven communication with retries helps services eventually sync data despite temporary failures.

3. Final Answer:

   Implementing event-driven communication with retries -> Option D

4. Quick Check:

   Events + retries = eventual consistency [OK]

Hint: Events and retries fix temporary inconsistency [OK]

Common Mistakes:

• Thinking synchronous locks work well across distributed services
• Assuming one shared database solves all consistency issues
• Disabling retries causes data loss, not consistency

3. Consider two microservices A and B. Service A updates data and sends an event to B. If B processes the event twice due to retry, what is the likely outcome?

medium

A. Data in B will be corrupted due to duplicate updates

B. B will ignore the second event automatically

C. B will apply the update twice unless idempotency is implemented

D. Service A will rollback its update

Solution

1. Step 1: Understand event retries in microservices

   Retries can cause the same event to be processed multiple times by a service.

2. Step 2: Analyze effect without idempotency

   Without idempotency, processing the same event twice causes duplicate updates, leading to incorrect data.

3. Final Answer:

   B will apply the update twice unless idempotency is implemented -> Option C

4. Quick Check:

   Idempotency prevents duplicate effects [OK]

Hint: Without idempotency, retries cause duplicate updates [OK]

Common Mistakes:

• Assuming retries are always ignored automatically
• Thinking service A rolls back on B's retry
• Believing duplicate events never affect data

4. A microservice system uses events to sync data but sometimes data is inconsistent. Which fix addresses this problem?

medium

A. Add idempotent processing for events

B. Store all data in one shared database

C. Use synchronous calls instead of events

D. Remove retries to avoid duplicate events

Solution

1. Step 1: Identify cause of inconsistency

   Retries cause duplicate events, leading to inconsistent data if processing is not idempotent.

2. Step 2: Choose best fix

   Making event processing idempotent ensures duplicates do not corrupt data, fixing inconsistency.

3. Final Answer:

   Add idempotent processing for events -> Option A

4. Quick Check:

   Idempotency fixes duplicate event issues [OK]

Hint: Idempotency fixes duplicate event problems [OK]

Common Mistakes:

• Removing retries causes lost updates
• Switching to synchronous calls reduces scalability
• Using one database breaks microservices independence

5. You design a microservices system where Service A updates inventory and Service B updates orders. Both must stay consistent. Which approach best handles data consistency challenges?

hard

A. Use distributed transactions with two-phase commit across services

B. Use event-driven architecture with eventual consistency and compensating actions

C. Store all data in a single monolithic database

D. Synchronously call Service B from Service A and block until done

Solution

1. Step 1: Understand distributed transaction challenges

   Two-phase commit is complex and reduces scalability in microservices.

2. Step 2: Evaluate event-driven eventual consistency

   Event-driven design with eventual consistency and compensating actions handles failures gracefully and scales well.

3. Step 3: Compare other options

   Monolithic DB breaks microservices independence; synchronous blocking reduces performance.

4. Final Answer:

   Use event-driven architecture with eventual consistency and compensating actions -> Option B

5. Quick Check:

   Event-driven + compensations = scalable consistency [OK]

Hint: Event-driven with compensations scales best for consistency [OK]

Common Mistakes:

• Choosing distributed transactions that hurt scalability
• Using monolithic DB breaks microservices benefits
• Blocking synchronous calls reduce system responsiveness