Eventual consistency handling in Microservices - System Design Exercise

Design: Eventual Consistency Handling in Microservices

Design focuses on data synchronization and consistency mechanisms between microservices. Does not cover UI design or specific business logic inside services.

Functional Requirements

FR1: Support multiple microservices that update shared data asynchronously

FR2: Ensure data changes propagate to all relevant services eventually

FR3: Allow services to continue operating with slightly stale data temporarily

FR4: Provide mechanisms to detect and resolve data conflicts

FR5: Maintain system availability and responsiveness during data synchronization

Non-Functional Requirements

NFR1: Handle up to 10,000 concurrent updates per minute

NFR2: Ensure eventual consistency within 5 minutes of an update

NFR3: API response latency p99 under 300ms for user-facing requests

NFR4: System availability target of 99.9% uptime

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Event/message broker (e.g., Kafka, RabbitMQ)

Change data capture or event sourcing mechanisms

Conflict resolution logic (e.g., last write wins, version vectors)

Service APIs with idempotency support

Monitoring and alerting for synchronization delays

Design Patterns

Event-driven architecture

Saga pattern for distributed transactions

CQRS (Command Query Responsibility Segregation)

Idempotent message processing

Versioning and vector clocks for conflict detection

Reference Architecture

  +----------------+       +----------------+       +----------------+
  |  Service A     |       |  Service B     |       |  Service C     |
  | (writes data)  |       | (reads & writes)|       | (reads data)   |
  +-------+--------+       +--------+-------+       +--------+-------+
          |                         |                        |
          |  Publish event/update   |                        |
          |------------------------>|                        |
          |                         |  Publish event/update   |
          |                         |------------------------>|
          |                         |                        |
          |                         |                        |
          |                         |                        |
  +-------v-------------------------v------------------------v-------+
  |                         Event Broker (Kafka)                      |
  +-------------------------------------------------------------------+
          |                         |                        |
          | Consume events           | Consume events         | Consume events
          |                         |                        |
  +-------v-------------------------v------------------------v-------+
  |  Service A local store  |  Service B local store  |  Service C local store |
  |  (eventual consistent)  |  (eventual consistent)  |  (eventual consistent) |
  +-------------------------+-------------------------+----------------+

Components

Service A, B, C

Any microservice framework (e.g., Spring Boot, Node.js, Go)

Business logic and local data storage; produce and consume events

Event Broker

Apache Kafka or RabbitMQ

Reliable event/message delivery between services for data synchronization

Local Data Store

Relational or NoSQL database per service

Store service-specific data with eventual consistency guarantees

Conflict Resolution Module

Custom logic in services

Detect and resolve conflicting updates using versioning or timestamps

Monitoring & Alerting

Prometheus, Grafana, or ELK stack

Track event lag, failures, and system health

Request Flow

1. 1. Service A updates its local data and publishes an event describing the change to the event broker.

2. 2. The event broker stores and distributes the event to subscribed services asynchronously.

3. 3. Service B and Service C consume the event and update their local data stores accordingly.

4. 4. If conflicting updates are detected (e.g., concurrent writes), services apply conflict resolution logic.

5. 5. Services continue to serve requests using their local data, which may be slightly stale until synchronization completes.

6. 6. Monitoring tools track event processing delays and alert if synchronization exceeds the 5-minute target.

Database Schema

Entities: DataRecord { id (PK), value, version, last_updated } Relationships: Each service maintains its own DataRecord table. Version field helps detect conflicts. No direct cross-service foreign keys due to decoupling.

Scaling Discussion

Bottlenecks

Event broker throughput limits when handling high update volumes

Increased event processing latency causing stale data beyond acceptable limits

Conflict resolution complexity grows with concurrent updates

Local data stores becoming bottlenecks under heavy read/write load

Solutions

Partition event topics and scale event broker cluster horizontally

Optimize consumer processing with parallelism and batching

Implement more sophisticated conflict resolution strategies or user intervention workflows

Use caching layers and database sharding to distribute load

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why eventual consistency is chosen over strong consistency for availability

Describe the role of the event broker in decoupling services

Discuss conflict detection and resolution approaches

Highlight monitoring importance to ensure synchronization health

Address scaling challenges and practical solutions