Eventual consistency handling in Microservices - Scalability & System Analysis

Scalability Analysis - Eventual consistency handling

Growth Table: Eventual Consistency Handling

Users/Traffic	What Changes?
100 users	Simple async messaging between services; low message volume; eventual consistency delays are minimal and unnoticeable.
10,000 users	Message queues grow; need for retry and dead-letter queues; monitoring of message lag; some delays in data sync become visible.
1,000,000 users	High message throughput; message brokers become bottleneck; need partitioning and scaling of queues; conflict resolution logic needed for data divergence.
100,000,000 users	Massive distributed messaging; multi-region replication; complex conflict resolution; eventual consistency delays impact user experience; advanced monitoring and alerting required.

First Bottleneck

The message broker or event queue becomes the first bottleneck as message volume grows. It can get overwhelmed by high throughput, causing delays and message backlogs. This slows down data synchronization between microservices, increasing eventual consistency delays.

Scaling Solutions

• Horizontal scaling: Add more instances of message brokers and partition topics to distribute load.
• Sharding: Partition data and messages by key to reduce contention and improve parallelism.
• Caching: Use caches to serve read requests quickly while waiting for eventual consistency.
• Idempotency and retries: Implement idempotent consumers and retry mechanisms to handle failures gracefully.
• Conflict resolution: Use versioning, timestamps, or CRDTs (Conflict-free Replicated Data Types) to resolve data conflicts.
• Monitoring and alerting: Track message lag, queue sizes, and processing times to detect bottlenecks early.
• Multi-region replication: Deploy brokers and services closer to users to reduce latency.

Back-of-Envelope Cost Analysis

• At 1M users, assume 10 requests per user per minute = ~166,000 requests/sec.
• Each request may generate 1-3 messages; message broker must handle ~500,000 messages/sec.
• Single Kafka broker can handle ~100,000 messages/sec; need ~5 brokers with partitioning.
• Storage for event logs grows rapidly; plan for terabytes per day depending on message size.
• Network bandwidth must support message replication; 1 Gbps link ~125 MB/s; plan multiple links or cloud bandwidth.

Interview Tip

Start by explaining what eventual consistency means in microservices. Then identify the main bottleneck (message broker). Discuss how message volume grows with users and how that affects latency. Propose scaling solutions like partitioning, retries, and conflict resolution. Finally, mention monitoring and user experience trade-offs.

Self Check

Your database handles 1000 QPS. Traffic grows 10x. What do you do first?

Answer: Since traffic increased to 10,000 QPS, the database is likely the bottleneck. First, add read replicas to distribute read load and implement caching to reduce direct database queries. Also, consider optimizing queries and connection pooling before scaling vertically or sharding.

Key Result

Message brokers become the first bottleneck as traffic grows; scaling requires partitioning, horizontal scaling, and robust conflict resolution to maintain eventual consistency.

Practice

1. What does eventual consistency mean in microservices?

easy

A. Services use a single database to avoid inconsistencies

B. All services update data instantly and always stay in sync

C. Data updates may be delayed but will become consistent over time

D. Data is never synchronized between services

Solution

1. Step 1: Understand the concept of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate and become consistent eventually.

2. Step 2: Compare options with the concept

   Only Data updates may be delayed but will become consistent over time describes delayed but eventual synchronization, matching the definition.

3. Final Answer:

   Data updates may be delayed but will become consistent over time -> Option C

4. Quick Check:

   Eventual consistency = delayed sync but consistent later [OK]

Hint: Look for delayed but guaranteed data sync over time [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming single database means eventual consistency

2. Which of the following is a correct way to handle eventual consistency in microservices?

easy

A. Use asynchronous event messages to update other services

B. Use synchronous calls between services for every update

C. Block user requests until all services are updated

D. Store all data in a single monolithic database

Solution

1. Step 1: Identify the correct communication pattern for eventual consistency

   Eventual consistency relies on asynchronous events to propagate updates without blocking.

2. Step 2: Evaluate options

   Use asynchronous event messages to update other services uses asynchronous event messages, which fits eventual consistency best.

3. Final Answer:

   Use asynchronous event messages to update other services -> Option A

4. Quick Check:

   Asynchronous events = eventual consistency [OK]

Hint: Choose asynchronous event-driven updates, not synchronous calls [OK]

Common Mistakes:

• Choosing synchronous calls which block and reduce scalability
• Thinking blocking user requests is needed
• Assuming monolithic DB solves consistency

3. Consider this simplified event processing code snippet in a microservice:

eventQueue = []

function processEvent(event) {
  if (event.type === 'update') {
    database.update(event.data)
  }
}

// Events arrive asynchronously
processEvent({type: 'update', data: {id: 1, value: 'A'}})
processEvent({type: 'update', data: {id: 1, value: 'B'}})

// What is the likely final value in the database for id 1?

medium

A. The value remains unchanged

B. 'A', because the first event updates the value

C. An error occurs due to conflicting updates

D. 'B', because the second event overwrites the first

Solution

1. Step 1: Analyze event processing order

   Events are processed in order: first update to 'A', then update to 'B'.

2. Step 2: Determine final database state

   The second update overwrites the first, so final value is 'B'.

3. Final Answer:

   'B', because the second event overwrites the first -> Option D

4. Quick Check:

   Last update wins = 'B' [OK]

Hint: Last event update overwrites previous data [OK]

Common Mistakes:

• Assuming first update persists ignoring later events
• Expecting errors on normal overwrites
• Thinking data stays unchanged without updates

4. A microservice uses event-driven updates but sometimes data conflicts occur. Which fix improves eventual consistency handling?

function handleEvent(event) {
  if (event.type === 'update') {
    if (!database.has(event.data.id)) {
      database.insert(event.data)
    } else {
      database.update(event.data)
    }
  }
}

medium

A. Add version numbers to events and apply only newer versions

B. Remove the check and always insert data

C. Process events synchronously to avoid conflicts

D. Ignore conflicting events silently

Solution

1. Step 1: Identify cause of conflicts

   Conflicts arise when updates arrive out of order or duplicate events occur.

2. Step 2: Apply versioning to resolve conflicts

   Using version numbers lets the service apply only the latest update, ensuring consistency.

3. Final Answer:

   Add version numbers to events and apply only newer versions -> Option A

4. Quick Check:

   Versioning resolves conflicts in eventual consistency [OK]

Hint: Use version numbers to apply only latest updates [OK]

Common Mistakes:

• Removing checks causes duplicate inserts
• Synchronous processing reduces scalability
• Ignoring conflicts leads to stale data

5. You design a microservices system where orders and inventory are separate services. To handle eventual consistency, which approach best ensures inventory updates reflect orders correctly despite delays?

hard

A. Store orders and inventory in the same database to avoid syncing

B. Use an event log where order service emits events and inventory service processes them asynchronously with retries

C. Make inventory service call order service synchronously for every update

D. Ignore inventory updates until orders are fully processed

Solution

1. Step 1: Understand the need for asynchronous communication

   Orders and inventory are separate; syncing asynchronously avoids blocking and scales better.

2. Step 2: Choose event log with retries for reliability

   Using an event log lets inventory process order events reliably, handling delays and retries to ensure consistency.

3. Final Answer:

   Use an event log where order service emits events and inventory service processes them asynchronously with retries -> Option B

4. Quick Check:

   Event log + async processing = robust eventual consistency [OK]

Hint: Use event logs with retries for reliable async sync [OK]

Common Mistakes:

• Using synchronous calls causing blocking
• Single database reduces microservices benefits
• Ignoring updates causes stale inventory