Lessons from microservices failures - Architecture Diagram

System Overview - Lessons from microservices failures

This system illustrates a typical microservices architecture and highlights common failure points. It shows how services communicate through APIs, use databases and caches, and handle asynchronous tasks with message queues. The key lesson is to understand failure impacts and mitigation strategies to build resilient microservices.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  +-----------------------------+
  |                             |
  v                             v
Service A                    Service B
  |                             |
  v                             v
Cache A                      Cache B
  |                             |
  v                             v
Database A                  Database B
  |
  v
Message Queue
  |
  v
Service C (Async Worker)

Components

User

client

Initiates requests to the system

Load Balancer

load_balancer

Distributes incoming traffic evenly to API Gateway instances

API Gateway

api_gateway

Routes requests to appropriate microservices and handles authentication

Service A

service

Handles business logic for feature A

Service B

service

Handles business logic for feature B

Cache A

cache

Stores frequently accessed data for Service A to reduce database load

Cache B

cache

Stores frequently accessed data for Service B to reduce database load

Database A

database

Persistent storage for Service A data

Database B

database

Persistent storage for Service B data

Message Queue

queue

Manages asynchronous communication and task processing

Service C (Async Worker)

service

Processes background tasks asynchronously from the queue

Request Flow - 15 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Service A

Service A → Cache A

Cache A → Service A

Service A → Database A

Database A → Service A

Service A → Cache A

Service A → Message Queue

Message Queue → Service C (Async Worker)

Service C (Async Worker) → Database B

Service C (Async Worker) → Cache B

Service A → API Gateway

API Gateway → Load Balancer

Load Balancer → User

Failure Scenario

Component Fails:Database A

Impact:Service A cannot write or read fresh data; cache may serve stale data; async tasks dependent on fresh data may fail or delay

Mitigation:Use database replication for failover; implement circuit breakers in Service A to degrade gracefully; rely on cache for reads temporarily; alert and auto-scale recovery processes

Architecture Quiz - 3 Questions

Which component ensures that user requests are evenly distributed to prevent overload?

A Load Balancer

B API Gateway

C Cache

D Message Queue

Design Principle

This architecture demonstrates the importance of separating synchronous and asynchronous processing, using caches to reduce database load, and employing load balancers and API gateways to manage traffic. Understanding failure impacts and mitigation strategies like replication, circuit breakers, and graceful degradation is key to building resilient microservices.

Practice

1. Which of the following is a key lesson from microservices failures to improve system resilience?

easy

A. Design services to be loosely coupled and handle failures gracefully

B. Combine all services into a single monolith to avoid communication issues

C. Ignore monitoring since failures are rare and unpredictable

D. Avoid retries to prevent additional load on services

Solution

1. Step 1: Understand microservices failure causes

   Failures often happen due to tight coupling and lack of fault tolerance.

2. Step 2: Identify best practice for resilience

   Loose coupling and graceful failure handling improve system stability.

3. Final Answer:

   Design services to be loosely coupled and handle failures gracefully -> Option A

4. Quick Check:

   Loose coupling = resilience [OK]

Hint: Remember: loose coupling prevents cascading failures [OK]

Common Mistakes:

• Thinking monoliths avoid failures
• Ignoring monitoring importance
• Avoiding retries completely

2. Which syntax correctly represents a retry mechanism with a limit in a microservice call?

easy

A. while(true) { callService() }

B. retry(count=-1) { callService() }

C. retry(0) { callService() }

D. retry(count=5) { callService() }

Solution

1. Step 1: Understand retry syntax with limits

   Retries must have a positive count to limit attempts.

2. Step 2: Evaluate options

   retry(count=5) { callService() } uses a positive count (5), valid retry limit; others are infinite or zero retries.

3. Final Answer:

   retry(count=5) { callService() } -> Option D

4. Quick Check:

   Positive retry count = correct syntax [OK]

Hint: Retries need a positive count to avoid infinite loops [OK]

Common Mistakes:

• Using infinite loops for retries
• Setting retry count to zero or negative
• Ignoring retry limits

3. Given this pseudocode for a microservice call with fallback:

result = callService() or fallbackService()

What will be the output if callService() fails but fallbackService() succeeds?

medium

A. An error is thrown and no result is returned

B. The result from callService() is returned despite failure

C. The result from fallbackService() is returned

D. Both results are combined and returned

Solution

1. Step 1: Understand fallback behavior

   If the main service fails, fallback is called to provide a result.

2. Step 2: Analyze given code

   Since callService() fails, fallbackService() result is used.

3. Final Answer:

   The result from fallbackService() is returned -> Option C

4. Quick Check:

   Fallback returns result on failure [OK]

Hint: Fallback runs only if main service fails [OK]

Common Mistakes:

• Assuming error is thrown without fallback
• Thinking main service result returns despite failure
• Believing results combine automatically

4. A microservice call retries 3 times on failure but never succeeds. What is the main issue in this retry design?

medium

A. No fallback mechanism to handle persistent failure

B. Retries cause infinite loops without limits

C. Retries are too few to recover from failure

D. Service calls are synchronous causing delays

Solution

1. Step 1: Analyze retry behavior

   Retries are limited to 3 attempts, so no infinite loop.

2. Step 2: Identify missing resilience feature

   Without fallback, system cannot recover after retries fail.

3. Final Answer:

   No fallback mechanism to handle persistent failure -> Option A

4. Quick Check:

   Retries need fallback for persistent failures [OK]

Hint: Retries alone can't fix persistent failures; add fallback [OK]

Common Mistakes:

• Confusing retry limits with infinite loops
• Assuming more retries always solve failures
• Ignoring fallback importance

5. You design a microservices system where Service A calls Service B, which calls Service C. Service C is unstable and often fails. Which design improves overall system stability best?

hard

A. Make Service A call Service C directly to reduce hops

B. Add retries with limits and fallback in Service B for calls to Service C

C. Remove retries to avoid extra load on Service C

D. Combine Services B and C into one to avoid network calls

Solution

1. Step 1: Identify failure point and impact

   Service C is unstable, causing failures in the chain.

2. Step 2: Apply fault tolerance best practices

   Retries with limits and fallback in Service B isolate failures and improve stability.

3. Step 3: Evaluate other options

   Direct calls or combining services increase coupling or load; removing retries loses resilience.

4. Final Answer:

   Add retries with limits and fallback in Service B for calls to Service C -> Option B

5. Quick Check:

   Retries + fallback near failure = stability [OK]

Hint: Place retries and fallback close to unstable service [OK]

Common Mistakes:

• Increasing coupling by combining services
• Bypassing intermediate services causing tight coupling
• Removing retries losing fault tolerance