Eventual consistency handling in Microservices - Architecture Diagram

System Overview - Eventual consistency handling

This system manages data updates across multiple microservices that do not update their data stores instantly. It ensures that all services eventually have consistent data by using asynchronous communication and retries.

Key requirements include handling temporary data mismatches gracefully and ensuring data synchronization without blocking user requests.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+----------------+       +----------------+       +----------------+
|  Service A     |<----->|  Message Queue |<----->|  Service B     |
| (Command)      |       | (Event Broker) |       | (Query)        |
+----------------+       +----------------+       +----------------+
       |                                              |
       v                                              v
+----------------+                             +----------------+
| Database A     |                             | Database B     |
+----------------+                             +----------------+

Components

User

user

Initiates requests to the system

Load Balancer

load_balancer

Distributes incoming requests evenly to API Gateway instances

API Gateway

api_gateway

Routes requests to appropriate microservices

Service A

service

Handles commands and writes data to Database A

Message Queue

message_queue

Asynchronously transfers events from Service A to Service B

Service B

service

Consumes events and updates Database B to reflect changes

Database A

database

Stores authoritative data for Service A

Database B

database

Stores data for Service B, eventually consistent with Database A

Request Flow - 8 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Service A

Service A → Database A

Service A → Message Queue

Message Queue → Service B

Service B → Database B

Service B → API Gateway

Failure Scenario

Component Fails:Message Queue

Impact:Events are delayed or lost, causing Service B's data to be stale and inconsistent with Service A

Mitigation:Use durable queues with retry and dead-letter mechanisms to ensure eventual delivery; monitor queue health and alert on failures

Architecture Quiz - 3 Questions

Which component ensures that Service B eventually receives updates from Service A?

A Load Balancer

B API Gateway

C Message Queue

D Database A

Design Principle

This architecture demonstrates the principle of eventual consistency by decoupling services with asynchronous messaging. It allows services to operate independently and update their data stores at different times while ensuring data synchronization over time.

Practice

1. What does eventual consistency mean in microservices?

easy

A. Services use a single database to avoid inconsistencies

B. All services update data instantly and always stay in sync

C. Data updates may be delayed but will become consistent over time

D. Data is never synchronized between services

Solution

1. Step 1: Understand the concept of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate and become consistent eventually.

2. Step 2: Compare options with the concept

   Only Data updates may be delayed but will become consistent over time describes delayed but eventual synchronization, matching the definition.

3. Final Answer:

   Data updates may be delayed but will become consistent over time -> Option C

4. Quick Check:

   Eventual consistency = delayed sync but consistent later [OK]

Hint: Look for delayed but guaranteed data sync over time [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming single database means eventual consistency

2. Which of the following is a correct way to handle eventual consistency in microservices?

easy

A. Use asynchronous event messages to update other services

B. Use synchronous calls between services for every update

C. Block user requests until all services are updated

D. Store all data in a single monolithic database

Solution

1. Step 1: Identify the correct communication pattern for eventual consistency

   Eventual consistency relies on asynchronous events to propagate updates without blocking.

2. Step 2: Evaluate options

   Use asynchronous event messages to update other services uses asynchronous event messages, which fits eventual consistency best.

3. Final Answer:

   Use asynchronous event messages to update other services -> Option A

4. Quick Check:

   Asynchronous events = eventual consistency [OK]

Hint: Choose asynchronous event-driven updates, not synchronous calls [OK]

Common Mistakes:

• Choosing synchronous calls which block and reduce scalability
• Thinking blocking user requests is needed
• Assuming monolithic DB solves consistency

3. Consider this simplified event processing code snippet in a microservice:

eventQueue = []

function processEvent(event) {
  if (event.type === 'update') {
    database.update(event.data)
  }
}

// Events arrive asynchronously
processEvent({type: 'update', data: {id: 1, value: 'A'}})
processEvent({type: 'update', data: {id: 1, value: 'B'}})

// What is the likely final value in the database for id 1?

medium

A. The value remains unchanged

B. 'A', because the first event updates the value

C. An error occurs due to conflicting updates

D. 'B', because the second event overwrites the first

Solution

1. Step 1: Analyze event processing order

   Events are processed in order: first update to 'A', then update to 'B'.

2. Step 2: Determine final database state

   The second update overwrites the first, so final value is 'B'.

3. Final Answer:

   'B', because the second event overwrites the first -> Option D

4. Quick Check:

   Last update wins = 'B' [OK]

Hint: Last event update overwrites previous data [OK]

Common Mistakes:

• Assuming first update persists ignoring later events
• Expecting errors on normal overwrites
• Thinking data stays unchanged without updates

4. A microservice uses event-driven updates but sometimes data conflicts occur. Which fix improves eventual consistency handling?

function handleEvent(event) {
  if (event.type === 'update') {
    if (!database.has(event.data.id)) {
      database.insert(event.data)
    } else {
      database.update(event.data)
    }
  }
}

medium

A. Add version numbers to events and apply only newer versions

B. Remove the check and always insert data

C. Process events synchronously to avoid conflicts

D. Ignore conflicting events silently

Solution

1. Step 1: Identify cause of conflicts

   Conflicts arise when updates arrive out of order or duplicate events occur.

2. Step 2: Apply versioning to resolve conflicts

   Using version numbers lets the service apply only the latest update, ensuring consistency.

3. Final Answer:

   Add version numbers to events and apply only newer versions -> Option A

4. Quick Check:

   Versioning resolves conflicts in eventual consistency [OK]

Hint: Use version numbers to apply only latest updates [OK]

Common Mistakes:

• Removing checks causes duplicate inserts
• Synchronous processing reduces scalability
• Ignoring conflicts leads to stale data

5. You design a microservices system where orders and inventory are separate services. To handle eventual consistency, which approach best ensures inventory updates reflect orders correctly despite delays?

hard

A. Store orders and inventory in the same database to avoid syncing

B. Use an event log where order service emits events and inventory service processes them asynchronously with retries

C. Make inventory service call order service synchronously for every update

D. Ignore inventory updates until orders are fully processed

Solution

1. Step 1: Understand the need for asynchronous communication

   Orders and inventory are separate; syncing asynchronously avoids blocking and scales better.

2. Step 2: Choose event log with retries for reliability

   Using an event log lets inventory process order events reliably, handling delays and retries to ensure consistency.

3. Final Answer:

   Use an event log where order service emits events and inventory service processes them asynchronously with retries -> Option B

4. Quick Check:

   Event log + async processing = robust eventual consistency [OK]

Hint: Use event logs with retries for reliable async sync [OK]

Common Mistakes:

• Using synchronous calls causing blocking
• Single database reduces microservices benefits
• Ignoring updates causes stale inventory