Eventual consistency in Microservices - Scalability & System Analysis

Scalability Analysis - Eventual consistency

Growth Table: Eventual Consistency in Microservices

Users/Traffic	What Changes?
100 users	Simple async messaging; low message volume; delays negligible; single message broker sufficient
10,000 users	Message volume grows; message broker load increases; need for partitioned topics/queues; slight delays in data sync appear
1,000,000 users	High message throughput; brokers need clustering; message ordering challenges; increased eventual consistency delays; monitoring critical
100,000,000 users	Massive message volume; multi-region brokers; complex partitioning and replication; network latency impacts consistency; advanced conflict resolution needed

First Bottleneck

The message broker or event bus is the first bottleneck. As user traffic grows, the volume of messages between microservices increases rapidly. A single broker instance can only handle so many messages per second before latency rises and messages queue up. This delays data synchronization and increases the time until all services reach consistency.

Scaling Solutions

• Horizontal scaling: Add more broker nodes to form a cluster, distributing message load and increasing throughput.
• Partitioning: Split topics or queues by key or service to parallelize message processing.
• Caching: Use local caches in services to reduce read load and tolerate stale data temporarily.
• Conflict resolution: Implement idempotent consumers and versioning to handle out-of-order or duplicate messages.
• Multi-region replication: Deploy brokers in multiple regions to reduce latency and improve availability.
• Monitoring and alerting: Track message lag and broker health to react before delays impact users.

Back-of-Envelope Cost Analysis

Assuming 1 million users generate 10 messages per second on average:

• Message rate: 10 million messages/sec
• Broker capacity: A single Kafka broker can handle ~100K-200K messages/sec, so ~50-100 brokers needed
• Storage: Messages stored temporarily; with 1KB per message, 10GB per second of data inflow
• Network bandwidth: 10 million messages * 1KB = ~10GB/s, requiring high bandwidth infrastructure

Interview Tip

When discussing eventual consistency scalability, start by explaining the trade-off between consistency and availability. Then identify the message broker as the main bottleneck. Discuss how message volume grows with users and how partitioning and clustering help. Mention the importance of monitoring and conflict resolution. Finally, highlight real-world challenges like network latency and multi-region setups.

Self Check

Your database handles 1000 QPS. Traffic grows 10x. What do you do first?

Answer: Since the database is the bottleneck, first add read replicas to distribute read load and implement caching to reduce direct database queries. For writes, consider queueing writes asynchronously or sharding data to scale write capacity.

Key Result

Eventual consistency scales by distributing message load across clustered brokers and partitioned queues, but the message broker is the first bottleneck as traffic grows.

Practice

1. What does eventual consistency mean in microservices?

easy

A. Data updates will be visible to all parts of the system after some delay

B. Data is always instantly consistent across all services

C. Data is never synchronized between services

D. Data updates happen only during system maintenance

Solution

1. Step 1: Understand the meaning of eventual consistency

   Eventual consistency means data changes are not immediate but will propagate over time.

2. Step 2: Compare options with the definition

   Only Data updates will be visible to all parts of the system after some delay correctly states that data updates become visible after some delay, matching eventual consistency.

3. Final Answer:

   Data updates will be visible to all parts of the system after some delay -> Option A

4. Quick Check:

   Eventual consistency = delayed data visibility [OK]

Hint: Eventual means "eventually", not instantly [OK]

Common Mistakes:

• Confusing eventual consistency with immediate consistency
• Thinking data never syncs
• Assuming updates only during maintenance

2. Which of the following is a correct way to implement eventual consistency in microservices?

easy

A. Use synchronous HTTP calls between services for every update

B. Use asynchronous event messaging to propagate changes

C. Block all reads until all writes complete

D. Disable communication between services

Solution

1. Step 1: Identify communication style for eventual consistency

   Eventual consistency relies on asynchronous communication to allow updates to propagate over time.

2. Step 2: Evaluate options

   Only Use asynchronous event messaging to propagate changes uses asynchronous event messaging, which fits eventual consistency. Others use synchronous or block reads, which do not.

3. Final Answer:

   Use asynchronous event messaging to propagate changes -> Option B

4. Quick Check:

   Asynchronous messaging = eventual consistency [OK]

Hint: Eventual consistency needs async events, not sync calls [OK]

Common Mistakes:

• Choosing synchronous calls which block updates
• Blocking reads causing poor availability
• Ignoring communication between services

3. Consider a microservice system where Service A updates data and publishes an event. Service B listens and updates its copy asynchronously. What is the expected state of Service B immediately after Service A's update?

medium

A. Service B has stale data until it processes the event

B. Service B rejects the update

C. Service B has the updated data instantly

D. Service B crashes due to inconsistency

Solution

1. Step 1: Understand asynchronous event propagation

   Service B updates data only after receiving and processing the event from Service A, which takes time.

2. Step 2: Determine Service B's state immediately after Service A's update

   Since event processing is asynchronous, Service B still holds old data until it processes the event.

3. Final Answer:

   Service B has stale data until it processes the event -> Option A

4. Quick Check:

   Async update means stale data initially [OK]

Hint: Async updates cause temporary stale data [OK]

Common Mistakes:

• Assuming instant data sync
• Thinking services reject updates
• Believing system crashes on inconsistency

4. A microservice system uses event-driven updates but sometimes Service B never receives events from Service A, causing stale data. What is the best fix?

medium

A. Switch to synchronous calls only

B. Ignore the problem as eventual consistency tolerates it

C. Implement event retry and dead-letter queues

D. Stop Service B from reading data

Solution

1. Step 1: Identify problem cause

   Missing events cause stale data because messages are lost or not delivered.

2. Step 2: Choose solution to ensure event delivery

   Implementing retries and dead-letter queues helps guarantee events reach Service B or are logged for manual handling.

3. Final Answer:

   Implement event retry and dead-letter queues -> Option C

4. Quick Check:

   Retries fix lost events = better consistency [OK]

Hint: Use retries and dead-letter queues for reliable events [OK]

Common Mistakes:

• Switching to sync calls losing scalability
• Ignoring lost events causing stale data
• Disabling reads instead of fixing events

5. You design a microservices system with eventual consistency. Service A updates inventory and publishes events. Service B updates order status based on inventory events. How do you ensure order status eventually matches inventory without blocking user requests?

hard

A. Store all data in a single database to avoid events

B. Make Service B synchronously call Service A for every order update

C. Block user requests until all services are consistent

D. Use asynchronous event processing with idempotent handlers and retries

Solution

1. Step 1: Understand requirements for eventual consistency and availability

   The system must update order status eventually without blocking user requests, so async processing is needed.

2. Step 2: Choose design that supports async updates safely

   Using asynchronous event processing with idempotent handlers and retries ensures updates happen reliably and without blocking.

3. Step 3: Evaluate other options

   Synchronous calls or blocking requests reduce availability; single database removes microservices benefits.

4. Final Answer:

   Use asynchronous event processing with idempotent handlers and retries -> Option D

5. Quick Check:

   Async + idempotent + retries = safe eventual consistency [OK]

Hint: Async with retries and idempotency ensures safe updates [OK]

Common Mistakes:

• Blocking user requests hurting availability
• Using sync calls causing tight coupling
• Ignoring idempotency causing duplicate updates