Eventual consistency handling in Microservices - System Design Exercise

Design: Eventual Consistency Handling in Microservices

Design focuses on data synchronization and consistency mechanisms between microservices. Does not cover UI design or specific business logic inside services.

Functional Requirements

FR1: Support multiple microservices that update shared data asynchronously

FR2: Ensure data changes propagate to all relevant services eventually

FR3: Allow services to continue operating with slightly stale data temporarily

FR4: Provide mechanisms to detect and resolve data conflicts

FR5: Maintain system availability and responsiveness during data synchronization

Non-Functional Requirements

NFR1: Handle up to 10,000 concurrent updates per minute

NFR2: Ensure eventual consistency within 5 minutes of an update

NFR3: API response latency p99 under 300ms for user-facing requests

NFR4: System availability target of 99.9% uptime

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Event/message broker (e.g., Kafka, RabbitMQ)

Change data capture or event sourcing mechanisms

Conflict resolution logic (e.g., last write wins, version vectors)

Service APIs with idempotency support

Monitoring and alerting for synchronization delays

Design Patterns

Event-driven architecture

Saga pattern for distributed transactions

CQRS (Command Query Responsibility Segregation)

Idempotent message processing

Versioning and vector clocks for conflict detection

Reference Architecture

  +----------------+       +----------------+       +----------------+
  |  Service A     |       |  Service B     |       |  Service C     |
  | (writes data)  |       | (reads & writes)|       | (reads data)   |
  +-------+--------+       +--------+-------+       +--------+-------+
          |                         |                        |
          |  Publish event/update   |                        |
          |------------------------>|                        |
          |                         |  Publish event/update   |
          |                         |------------------------>|
          |                         |                        |
          |                         |                        |
          |                         |                        |
  +-------v-------------------------v------------------------v-------+
  |                         Event Broker (Kafka)                      |
  +-------------------------------------------------------------------+
          |                         |                        |
          | Consume events           | Consume events         | Consume events
          |                         |                        |
  +-------v-------------------------v------------------------v-------+
  |  Service A local store  |  Service B local store  |  Service C local store |
  |  (eventual consistent)  |  (eventual consistent)  |  (eventual consistent) |
  +-------------------------+-------------------------+----------------+

Components

Service A, B, C

Any microservice framework (e.g., Spring Boot, Node.js, Go)

Business logic and local data storage; produce and consume events

Event Broker

Apache Kafka or RabbitMQ

Reliable event/message delivery between services for data synchronization

Local Data Store

Relational or NoSQL database per service

Store service-specific data with eventual consistency guarantees

Conflict Resolution Module

Custom logic in services

Detect and resolve conflicting updates using versioning or timestamps

Monitoring & Alerting

Prometheus, Grafana, or ELK stack

Track event lag, failures, and system health

Request Flow

1. 1. Service A updates its local data and publishes an event describing the change to the event broker.

2. 2. The event broker stores and distributes the event to subscribed services asynchronously.

3. 3. Service B and Service C consume the event and update their local data stores accordingly.

4. 4. If conflicting updates are detected (e.g., concurrent writes), services apply conflict resolution logic.

5. 5. Services continue to serve requests using their local data, which may be slightly stale until synchronization completes.

6. 6. Monitoring tools track event processing delays and alert if synchronization exceeds the 5-minute target.

Database Schema

Entities: DataRecord { id (PK), value, version, last_updated } Relationships: Each service maintains its own DataRecord table. Version field helps detect conflicts. No direct cross-service foreign keys due to decoupling.

Scaling Discussion

Bottlenecks

Event broker throughput limits when handling high update volumes

Increased event processing latency causing stale data beyond acceptable limits

Conflict resolution complexity grows with concurrent updates

Local data stores becoming bottlenecks under heavy read/write load

Solutions

Partition event topics and scale event broker cluster horizontally

Optimize consumer processing with parallelism and batching

Implement more sophisticated conflict resolution strategies or user intervention workflows

Use caching layers and database sharding to distribute load

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why eventual consistency is chosen over strong consistency for availability

Describe the role of the event broker in decoupling services

Discuss conflict detection and resolution approaches

Highlight monitoring importance to ensure synchronization health

Address scaling challenges and practical solutions

Practice

1. What does eventual consistency mean in microservices?

easy

A. Services use a single database to avoid inconsistencies

B. All services update data instantly and always stay in sync

C. Data updates may be delayed but will become consistent over time

D. Data is never synchronized between services

Solution

1. Step 1: Understand the concept of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate and become consistent eventually.

2. Step 2: Compare options with the concept

   Only Data updates may be delayed but will become consistent over time describes delayed but eventual synchronization, matching the definition.

3. Final Answer:

   Data updates may be delayed but will become consistent over time -> Option C

4. Quick Check:

   Eventual consistency = delayed sync but consistent later [OK]

Hint: Look for delayed but guaranteed data sync over time [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming single database means eventual consistency

2. Which of the following is a correct way to handle eventual consistency in microservices?

easy

A. Use asynchronous event messages to update other services

B. Use synchronous calls between services for every update

C. Block user requests until all services are updated

D. Store all data in a single monolithic database

Solution

1. Step 1: Identify the correct communication pattern for eventual consistency

   Eventual consistency relies on asynchronous events to propagate updates without blocking.

2. Step 2: Evaluate options

   Use asynchronous event messages to update other services uses asynchronous event messages, which fits eventual consistency best.

3. Final Answer:

   Use asynchronous event messages to update other services -> Option A

4. Quick Check:

   Asynchronous events = eventual consistency [OK]

Hint: Choose asynchronous event-driven updates, not synchronous calls [OK]

Common Mistakes:

• Choosing synchronous calls which block and reduce scalability
• Thinking blocking user requests is needed
• Assuming monolithic DB solves consistency

3. Consider this simplified event processing code snippet in a microservice:

eventQueue = []

function processEvent(event) {
  if (event.type === 'update') {
    database.update(event.data)
  }
}

// Events arrive asynchronously
processEvent({type: 'update', data: {id: 1, value: 'A'}})
processEvent({type: 'update', data: {id: 1, value: 'B'}})

// What is the likely final value in the database for id 1?

medium

A. The value remains unchanged

B. 'A', because the first event updates the value

C. An error occurs due to conflicting updates

D. 'B', because the second event overwrites the first

Solution

1. Step 1: Analyze event processing order

   Events are processed in order: first update to 'A', then update to 'B'.

2. Step 2: Determine final database state

   The second update overwrites the first, so final value is 'B'.

3. Final Answer:

   'B', because the second event overwrites the first -> Option D

4. Quick Check:

   Last update wins = 'B' [OK]

Hint: Last event update overwrites previous data [OK]

Common Mistakes:

• Assuming first update persists ignoring later events
• Expecting errors on normal overwrites
• Thinking data stays unchanged without updates

4. A microservice uses event-driven updates but sometimes data conflicts occur. Which fix improves eventual consistency handling?

function handleEvent(event) {
  if (event.type === 'update') {
    if (!database.has(event.data.id)) {
      database.insert(event.data)
    } else {
      database.update(event.data)
    }
  }
}

medium

A. Add version numbers to events and apply only newer versions

B. Remove the check and always insert data

C. Process events synchronously to avoid conflicts

D. Ignore conflicting events silently

Solution

1. Step 1: Identify cause of conflicts

   Conflicts arise when updates arrive out of order or duplicate events occur.

2. Step 2: Apply versioning to resolve conflicts

   Using version numbers lets the service apply only the latest update, ensuring consistency.

3. Final Answer:

   Add version numbers to events and apply only newer versions -> Option A

4. Quick Check:

   Versioning resolves conflicts in eventual consistency [OK]

Hint: Use version numbers to apply only latest updates [OK]

Common Mistakes:

• Removing checks causes duplicate inserts
• Synchronous processing reduces scalability
• Ignoring conflicts leads to stale data

5. You design a microservices system where orders and inventory are separate services. To handle eventual consistency, which approach best ensures inventory updates reflect orders correctly despite delays?

hard

A. Store orders and inventory in the same database to avoid syncing

B. Use an event log where order service emits events and inventory service processes them asynchronously with retries

C. Make inventory service call order service synchronously for every update

D. Ignore inventory updates until orders are fully processed

Solution

1. Step 1: Understand the need for asynchronous communication

   Orders and inventory are separate; syncing asynchronously avoids blocking and scales better.

2. Step 2: Choose event log with retries for reliability

   Using an event log lets inventory process order events reliably, handling delays and retries to ensure consistency.

3. Final Answer:

   Use an event log where order service emits events and inventory service processes them asynchronously with retries -> Option B

4. Quick Check:

   Event log + async processing = robust eventual consistency [OK]

Hint: Use event logs with retries for reliable async sync [OK]

Common Mistakes:

• Using synchronous calls causing blocking
• Single database reduces microservices benefits
• Ignoring updates causes stale inventory