Eventual consistency handling in Microservices - System Design Guide

Problem Statement

When multiple microservices update shared data independently, immediate consistency is hard to guarantee. This causes temporary data mismatches, stale reads, or conflicting updates that confuse users and break workflows.

Solution

Eventual consistency handling allows services to update data asynchronously and propagate changes over time. Services communicate changes via events or messages, ensuring all parts eventually see the same data without blocking operations or requiring tight coordination.

Architecture

Service A

→Event Bus

↓

Database A

This diagram shows two microservices updating their own databases and communicating changes asynchronously through an event bus to achieve eventual consistency.

Trade-offs

✓ Pros

→

Improves system availability by avoiding synchronous blocking calls between services.

→

Allows services to operate independently, improving scalability and fault tolerance.

→

Reduces coupling between services, enabling easier maintenance and deployment.

✗ Cons

→

Data may be temporarily inconsistent, leading to stale reads or conflicts.

→

Requires complex conflict resolution and retry mechanisms.

→

Debugging and tracing data flow becomes harder due to asynchronous nature.

Use when your system has multiple distributed services that update shared data and can tolerate short periods of inconsistency, typically at scales above thousands of requests per second.

Avoid when your application requires strong consistency guarantees, such as financial transactions or critical inventory updates where stale data causes unacceptable errors.

Real World Examples

Amazon

Amazon uses eventual consistency in its order processing microservices to allow independent updates of inventory and payment services without blocking user experience.

Uber

Uber applies eventual consistency to synchronize ride status updates across driver and rider services, allowing high availability despite network delays.

Netflix

Netflix uses eventual consistency to update user preferences and viewing history asynchronously across multiple services to maintain responsiveness.

Code Example

The before code shows a synchronous call from OrderService to InventoryService, causing blocking and tight coupling. The after code publishes an event asynchronously, allowing InventoryService to update stock independently, enabling eventual consistency.

Microservices

### Before: Synchronous update causing blocking and tight coupling
class OrderService:
    def create_order(self, order_data):
        # Update order database
        self.db.save(order_data)
        # Synchronously call inventory service
        success = self.inventory_service.reserve_stock(order_data['items'])
        if not success:
            raise Exception('Stock reservation failed')


### After: Asynchronous event-driven update for eventual consistency
class OrderService:
    def create_order(self, order_data):
        # Update order database
        self.db.save(order_data)
        # Publish event to event bus
        event = {'type': 'OrderCreated', 'data': order_data}
        self.event_bus.publish(event)

class InventoryService:
    def on_order_created(self, event):
        items = event['data']['items']
        # Reserve stock asynchronously
        self.db.reserve_stock(items)

# Event bus delivers events asynchronously, decoupling services

Alternatives

Strong consistency

Requires synchronous coordination and locking to ensure all services see the same data immediately.

Use when: Choose when your system cannot tolerate any stale or conflicting data, such as banking or inventory control.

Two-phase commit

A distributed transaction protocol that locks resources and commits changes atomically across services.

Use when: Choose when you need atomic updates across multiple services but can tolerate higher latency and complexity.

Summary

Eventual consistency allows distributed services to update data asynchronously, avoiding blocking and tight coupling.

It improves availability and scalability but requires handling temporary data inconsistencies and conflicts.

Use it when your system can tolerate short delays in data synchronization and needs to scale across multiple services.

Practice

1. What does eventual consistency mean in microservices?

easy

A. Services use a single database to avoid inconsistencies

B. All services update data instantly and always stay in sync

C. Data updates may be delayed but will become consistent over time

D. Data is never synchronized between services

Solution

1. Step 1: Understand the concept of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate and become consistent eventually.

2. Step 2: Compare options with the concept

   Only Data updates may be delayed but will become consistent over time describes delayed but eventual synchronization, matching the definition.

3. Final Answer:

   Data updates may be delayed but will become consistent over time -> Option C

4. Quick Check:

   Eventual consistency = delayed sync but consistent later [OK]

Hint: Look for delayed but guaranteed data sync over time [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming single database means eventual consistency

2. Which of the following is a correct way to handle eventual consistency in microservices?

easy

A. Use asynchronous event messages to update other services

B. Use synchronous calls between services for every update

C. Block user requests until all services are updated

D. Store all data in a single monolithic database

Solution

1. Step 1: Identify the correct communication pattern for eventual consistency

   Eventual consistency relies on asynchronous events to propagate updates without blocking.

2. Step 2: Evaluate options

   Use asynchronous event messages to update other services uses asynchronous event messages, which fits eventual consistency best.

3. Final Answer:

   Use asynchronous event messages to update other services -> Option A

4. Quick Check:

   Asynchronous events = eventual consistency [OK]

Hint: Choose asynchronous event-driven updates, not synchronous calls [OK]

Common Mistakes:

• Choosing synchronous calls which block and reduce scalability
• Thinking blocking user requests is needed
• Assuming monolithic DB solves consistency

3. Consider this simplified event processing code snippet in a microservice:

eventQueue = []

function processEvent(event) {
  if (event.type === 'update') {
    database.update(event.data)
  }
}

// Events arrive asynchronously
processEvent({type: 'update', data: {id: 1, value: 'A'}})
processEvent({type: 'update', data: {id: 1, value: 'B'}})

// What is the likely final value in the database for id 1?

medium

A. The value remains unchanged

B. 'A', because the first event updates the value

C. An error occurs due to conflicting updates

D. 'B', because the second event overwrites the first

Solution

1. Step 1: Analyze event processing order

   Events are processed in order: first update to 'A', then update to 'B'.

2. Step 2: Determine final database state

   The second update overwrites the first, so final value is 'B'.

3. Final Answer:

   'B', because the second event overwrites the first -> Option D

4. Quick Check:

   Last update wins = 'B' [OK]

Hint: Last event update overwrites previous data [OK]

Common Mistakes:

• Assuming first update persists ignoring later events
• Expecting errors on normal overwrites
• Thinking data stays unchanged without updates

4. A microservice uses event-driven updates but sometimes data conflicts occur. Which fix improves eventual consistency handling?

function handleEvent(event) {
  if (event.type === 'update') {
    if (!database.has(event.data.id)) {
      database.insert(event.data)
    } else {
      database.update(event.data)
    }
  }
}

medium

A. Add version numbers to events and apply only newer versions

B. Remove the check and always insert data

C. Process events synchronously to avoid conflicts

D. Ignore conflicting events silently

Solution

1. Step 1: Identify cause of conflicts

   Conflicts arise when updates arrive out of order or duplicate events occur.

2. Step 2: Apply versioning to resolve conflicts

   Using version numbers lets the service apply only the latest update, ensuring consistency.

3. Final Answer:

   Add version numbers to events and apply only newer versions -> Option A

4. Quick Check:

   Versioning resolves conflicts in eventual consistency [OK]

Hint: Use version numbers to apply only latest updates [OK]

Common Mistakes:

• Removing checks causes duplicate inserts
• Synchronous processing reduces scalability
• Ignoring conflicts leads to stale data

5. You design a microservices system where orders and inventory are separate services. To handle eventual consistency, which approach best ensures inventory updates reflect orders correctly despite delays?

hard

A. Store orders and inventory in the same database to avoid syncing

B. Use an event log where order service emits events and inventory service processes them asynchronously with retries

C. Make inventory service call order service synchronously for every update

D. Ignore inventory updates until orders are fully processed

Solution

1. Step 1: Understand the need for asynchronous communication

   Orders and inventory are separate; syncing asynchronously avoids blocking and scales better.

2. Step 2: Choose event log with retries for reliability

   Using an event log lets inventory process order events reliably, handling delays and retries to ensure consistency.

3. Final Answer:

   Use an event log where order service emits events and inventory service processes them asynchronously with retries -> Option B

4. Quick Check:

   Event log + async processing = robust eventual consistency [OK]

Hint: Use event logs with retries for reliable async sync [OK]

Common Mistakes:

• Using synchronous calls causing blocking
• Single database reduces microservices benefits
• Ignoring updates causes stale inventory