Eventual consistency in Microservices - System Design Guide

Problem Statement

When multiple services update shared data independently, immediate synchronization can fail, causing temporary data conflicts and stale reads. This leads to user confusion and errors if the system expects all parts to have the exact same data instantly.

Solution

Eventual consistency allows services to update data independently and asynchronously, accepting temporary differences. Over time, updates propagate through the system until all services converge to the same data state, ensuring consistency without blocking operations.

Architecture

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Service A   │──────▶│ Event Bus   │──────▶│ Service B   │
└─────────────┘       └─────────────┘       └─────────────┘
       │                                         ▲
       │                                         │
       └─────────────────────────────────────────┘


Legend:
- Service A publishes updates to Event Bus.
- Service B consumes events and updates its data asynchronously.
- Data converges over time.

This diagram shows how services communicate updates via an event bus asynchronously, allowing data to become consistent eventually.

Trade-offs

✓ Pros

→

Improves system availability by avoiding synchronous locks or waits.

→

Enables high scalability as services operate independently.

→

Reduces latency for write operations since they don't wait for global consensus.

✗ Cons

→

Temporary data inconsistencies can confuse users or cause errors.

→

Requires complex conflict resolution and reconciliation logic.

→

Harder to reason about system state at any given moment.

Use when your system has high write throughput, distributed services, and can tolerate short periods of inconsistency, such as social media feeds or shopping carts.

Avoid when strong consistency is critical, like financial transactions or inventory counts where stale data can cause serious errors.

Real World Examples

Amazon

Uses eventual consistency in its DynamoDB to allow distributed data replication with high availability and partition tolerance.

Netflix

Applies eventual consistency in its microservices to handle user preferences and viewing history updates asynchronously for better scalability.

LinkedIn

Employs eventual consistency in its feed system to propagate updates across distributed services without blocking user interactions.

Code Example

This code shows how to move from blocking synchronous updates to asynchronous event-driven updates. The InventoryService updates its data and publishes an event without waiting for others. The NotificationService listens and updates itself later, allowing data to converge over time.

Microservices

### Before: Synchronous update causing blocking and tight coupling
class InventoryService:
    def update_stock(self, product_id, quantity):
        # Directly update database and block until done
        database.update(product_id, quantity)
        notify_other_services(product_id, quantity)


### After: Eventual consistency with asynchronous event publishing
import asyncio

class InventoryService:
    async def update_stock(self, product_id, quantity):
        # Update local database immediately
        database.update(product_id, quantity)
        # Publish event asynchronously
        await event_bus.publish('stock_updated', {'product_id': product_id, 'quantity': quantity})

class NotificationService:
    async def handle_stock_update(self, event):
        # Consume event and update own state asynchronously
        await database.update(event['product_id'], event['quantity'])

# Explanation:
# The before code blocks until all updates and notifications complete, causing delays.
# The after code updates local data immediately and publishes events asynchronously,
# allowing other services to update later, achieving eventual consistency.

Alternatives

Strong consistency

Requires all nodes to agree on data updates before confirming success, blocking operations until consensus.

Use when: When correctness and immediate data accuracy are critical, such as banking or inventory management.

Read-after-write consistency

Guarantees that a read immediately following a write returns the updated data, but may not scale well in distributed systems.

Use when: When users expect to see their own updates instantly but global consistency is less critical.

Summary

Eventual consistency allows distributed services to update data independently and asynchronously.

It improves availability and scalability by avoiding blocking operations.

Temporary inconsistencies require conflict resolution and are unsuitable for critical data.

Practice

1. What does eventual consistency mean in microservices?

easy

A. Data updates will be visible to all parts of the system after some delay

B. Data is always instantly consistent across all services

C. Data is never synchronized between services

D. Data updates happen only during system maintenance

Solution

1. Step 1: Understand the meaning of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate over time.

2. Step 2: Compare options with the definition

   Only Data updates will be visible to all parts of the system after some delay correctly states that data updates become visible after some delay, matching eventual consistency.

3. Final Answer:

   Data updates will be visible to all parts of the system after some delay -> Option A

4. Quick Check:

   Eventual consistency = delayed data visibility [OK]

Hint: Eventual means "eventually", not instantly [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming updates only during maintenance

2. Which of the following is a correct way to implement eventual consistency in microservices?

easy

A. Use synchronous HTTP calls between services for every update

B. Use asynchronous event messaging to propagate changes

C. Block all reads until all writes complete

D. Disable communication between services

Solution

1. Step 1: Identify communication style for eventual consistency

   Eventual consistency relies on asynchronous communication to allow updates to propagate over time.

2. Step 2: Evaluate options

   Only Use asynchronous event messaging to propagate changes uses asynchronous event messaging, which fits eventual consistency. Others use synchronous or block reads, which do not.

3. Final Answer:

   Use asynchronous event messaging to propagate changes -> Option B

4. Quick Check:

   Asynchronous messaging = eventual consistency [OK]

Hint: Eventual consistency needs async events, not sync calls [OK]

Common Mistakes:

• Choosing synchronous calls which block updates
• Blocking reads causing poor availability
• Ignoring communication between services

3. Consider a microservice system where Service A updates data and publishes an event. Service B listens and updates its copy asynchronously. What is the expected state of Service B immediately after Service A's update?

medium

A. Service B has stale data until it processes the event

B. Service B rejects the update

C. Service B has the updated data instantly

D. Service B crashes due to inconsistency

Solution

1. Step 1: Understand asynchronous event propagation

   Service B updates data only after receiving and processing the event from Service A, which takes time.

2. Step 2: Determine Service B's state immediately after Service A's update

   Since event processing is asynchronous, Service B still holds old data until it processes the event.

3. Final Answer:

   Service B has stale data until it processes the event -> Option A

4. Quick Check:

   Async update means stale data initially [OK]

Hint: Async updates cause temporary stale data [OK]

Common Mistakes:

• Assuming instant data sync
• Thinking services reject updates
• Believing system crashes on inconsistency

4. A microservice system uses event-driven updates but sometimes Service B never receives events from Service A, causing stale data. What is the best fix?

medium

A. Switch to synchronous calls only

B. Ignore the problem as eventual consistency tolerates it

C. Implement event retry and dead-letter queues

D. Stop Service B from reading data

Solution

1. Step 1: Identify problem cause

   Missing events cause stale data because messages are lost or not delivered.

2. Step 2: Choose solution to ensure event delivery

   Implementing retries and dead-letter queues helps guarantee events reach Service B or are logged for manual handling.

3. Final Answer:

   Implement event retry and dead-letter queues -> Option C

4. Quick Check:

   Retries fix lost events = better consistency [OK]

Hint: Use retries and dead-letter queues for reliable events [OK]

Common Mistakes:

• Switching to sync calls losing scalability
• Ignoring lost events causing stale data
• Disabling reads instead of fixing events

5. You design a microservices system with eventual consistency. Service A updates inventory and publishes events. Service B updates order status based on inventory events. How do you ensure order status eventually matches inventory without blocking user requests?

hard

A. Store all data in a single database to avoid events

B. Make Service B synchronously call Service A for every order update

C. Block user requests until all services are consistent

D. Use asynchronous event processing with idempotent handlers and retries

Solution

1. Step 1: Understand requirements for eventual consistency and availability

   The system must update order status eventually without blocking user requests, so async processing is needed.

2. Step 2: Choose design that supports async updates safely

   Using asynchronous event processing with idempotent handlers and retries ensures updates happen reliably and without blocking.

3. Step 3: Evaluate other options

   Synchronous calls or blocking requests reduce availability; single database removes microservices benefits.

4. Final Answer:

   Use asynchronous event processing with idempotent handlers and retries -> Option D

5. Quick Check:

   Async + idempotent + retries = safe eventual consistency [OK]

Hint: Async with retries and idempotency ensures safe updates [OK]

Common Mistakes:

• Blocking user requests hurting availability
• Using sync calls causing tight coupling
• Ignoring idempotency causing duplicate updates