Lessons from microservices failures - Scalability & System Analysis

Scalability Analysis - Lessons from microservices failures

Growth Table: Microservices Failures at Different Scales

Users / Traffic	Common Issues	System Behavior	Impact
100 users	Simple service communication, minor latency	Mostly stable, occasional slowdowns	Low impact, easy to debug
10,000 users	Increased network calls, partial failures, inconsistent data	Some services slow or fail, retries increase load	Noticeable user delays, error spikes
1,000,000 users	Service cascading failures, data inconsistency, deployment complexity	Frequent outages, degraded performance, hard to isolate faults	Major user impact, revenue loss
100,000,000 users	Global outages, complex dependency chains, monitoring overload	System-wide failures, slow recovery, high operational cost	Severe business impact, brand damage

First Bottleneck: Service Communication and Dependency Management

As microservices grow, the first bottleneck is the communication between services. Network latency and failures increase with more services and calls. Also, tightly coupled dependencies cause cascading failures when one service goes down. This breaks the system before hardware or database limits are reached.

Scaling Solutions for Microservices Failures

• Decouple services: Use asynchronous messaging and event-driven patterns to reduce tight coupling.
• Implement circuit breakers: Prevent cascading failures by stopping calls to failing services.
• Use service meshes: Manage communication, retries, and observability centrally.
• Improve monitoring and tracing: Detect failures early and understand dependencies.
• Automate deployments: Use canary releases and blue-green deployments to reduce risk.
• Scale horizontally: Add more instances of critical services to handle load.
• Cache responses: Reduce load on services by caching frequent data.

Back-of-Envelope Cost Analysis

• At 1M users, expect millions of inter-service calls per second, increasing network bandwidth and CPU usage.
• Storage needs grow for logs and tracing data; plan for terabytes daily.
• Monitoring and alerting systems must handle high data volumes, increasing operational costs.
• Horizontal scaling of services increases cloud compute costs linearly with traffic.

Interview Tip: Structuring Microservices Scalability Discussion

Start by identifying key components and their interactions. Discuss how communication patterns can cause bottlenecks. Explain failure modes like cascading failures and data inconsistency. Propose concrete solutions such as circuit breakers and asynchronous messaging. Highlight monitoring importance. Finally, consider cost and operational complexity as the system scales.

Self-Check Question

Your microservices system handles 1000 QPS. Traffic grows 10x. You notice increased latency and some service failures. What is your first action and why?

Key Result

Microservices systems first break due to increased inter-service communication and dependency failures as traffic grows; decoupling services and adding resilience patterns are key to scaling.

Practice

1. Which of the following is a key lesson from microservices failures to improve system resilience?

easy

A. Design services to be loosely coupled and handle failures gracefully

B. Combine all services into a single monolith to avoid communication issues

C. Ignore monitoring since failures are rare and unpredictable

D. Avoid retries to prevent additional load on services

Solution

1. Step 1: Understand microservices failure causes

   Failures often happen due to tight coupling and lack of fault tolerance.

2. Step 2: Identify best practice for resilience

   Loose coupling and graceful failure handling improve system stability.

3. Final Answer:

   Design services to be loosely coupled and handle failures gracefully -> Option A

4. Quick Check:

   Loose coupling = resilience [OK]

Hint: Remember: loose coupling prevents cascading failures [OK]

Common Mistakes:

• Thinking monoliths avoid failures
• Ignoring monitoring importance
• Avoiding retries completely

2. Which syntax correctly represents a retry mechanism with a limit in a microservice call?

easy

A. while(true) { callService() }

B. retry(count=-1) { callService() }

C. retry(0) { callService() }

D. retry(count=5) { callService() }

Solution

1. Step 1: Understand retry syntax with limits

   Retries must have a positive count to limit attempts.

2. Step 2: Evaluate options

   retry(count=5) { callService() } uses a positive count (5), valid retry limit; others are infinite or zero retries.

3. Final Answer:

   retry(count=5) { callService() } -> Option D

4. Quick Check:

   Positive retry count = correct syntax [OK]

Hint: Retries need a positive count to avoid infinite loops [OK]

Common Mistakes:

• Using infinite loops for retries
• Setting retry count to zero or negative
• Ignoring retry limits

3. Given this pseudocode for a microservice call with fallback:

result = callService() or fallbackService()

What will be the output if callService() fails but fallbackService() succeeds?

medium

A. An error is thrown and no result is returned

B. The result from callService() is returned despite failure

C. The result from fallbackService() is returned

D. Both results are combined and returned

Solution

1. Step 1: Understand fallback behavior

   If the main service fails, fallback is called to provide a result.

2. Step 2: Analyze given code

   Since callService() fails, fallbackService() result is used.

3. Final Answer:

   The result from fallbackService() is returned -> Option C

4. Quick Check:

   Fallback returns result on failure [OK]

Hint: Fallback runs only if main service fails [OK]

Common Mistakes:

• Assuming error is thrown without fallback
• Thinking main service result returns despite failure
• Believing results combine automatically

4. A microservice call retries 3 times on failure but never succeeds. What is the main issue in this retry design?

medium

A. No fallback mechanism to handle persistent failure

B. Retries cause infinite loops without limits

C. Retries are too few to recover from failure

D. Service calls are synchronous causing delays

Solution

1. Step 1: Analyze retry behavior

   Retries are limited to 3 attempts, so no infinite loop.

2. Step 2: Identify missing resilience feature

   Without fallback, system cannot recover after retries fail.

3. Final Answer:

   No fallback mechanism to handle persistent failure -> Option A

4. Quick Check:

   Retries need fallback for persistent failures [OK]

Hint: Retries alone can't fix persistent failures; add fallback [OK]

Common Mistakes:

• Confusing retry limits with infinite loops
• Assuming more retries always solve failures
• Ignoring fallback importance

5. You design a microservices system where Service A calls Service B, which calls Service C. Service C is unstable and often fails. Which design improves overall system stability best?

hard

A. Make Service A call Service C directly to reduce hops

B. Add retries with limits and fallback in Service B for calls to Service C

C. Remove retries to avoid extra load on Service C

D. Combine Services B and C into one to avoid network calls

Solution

1. Step 1: Identify failure point and impact

   Service C is unstable, causing failures in the chain.

2. Step 2: Apply fault tolerance best practices

   Retries with limits and fallback in Service B isolate failures and improve stability.

3. Step 3: Evaluate other options

   Direct calls or combining services increase coupling or load; removing retries loses resilience.

4. Final Answer:

   Add retries with limits and fallback in Service B for calls to Service C -> Option B

5. Quick Check:

   Retries + fallback near failure = stability [OK]

Hint: Place retries and fallback close to unstable service [OK]

Common Mistakes:

• Increasing coupling by combining services
• Bypassing intermediate services causing tight coupling
• Removing retries losing fault tolerance