Lessons from microservices failures - System Design Exercise

Design: Microservices Architecture Lessons

Focus on microservices failure causes and mitigation strategies including architecture, communication, data management, and deployment. Out of scope: detailed implementation of each microservice business logic.

Functional Requirements

FR1: Understand common failure points in microservices systems

FR2: Identify causes of failures such as cascading failures, data inconsistency, and deployment issues

FR3: Learn best practices to prevent or mitigate these failures

FR4: Design a resilient microservices system incorporating these lessons

Non-Functional Requirements

NFR1: System should handle 10,000 concurrent requests with p99 latency under 300ms

NFR2: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR3: Support eventual consistency where applicable

NFR4: Allow independent deployment of services without downtime

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

API Gateway

Service Registry and Discovery

Load Balancer

Circuit Breaker

Message Queue

Centralized Logging and Monitoring

Database per Service

Deployment Pipeline with Canary Releases

Design Patterns

Circuit Breaker Pattern

Bulkhead Isolation

Eventual Consistency with Event Sourcing

Retry with Exponential Backoff

Blue-Green and Canary Deployments

Centralized Logging and Distributed Tracing

Reference Architecture

                +--------------------+
                |    API Gateway     |
                +---------+----------+
                          |
          +---------------+----------------+
          |                                |
  +-------v-------+                +-------v-------+
  |  Service A    |                |  Service B    |
  +-------+-------+                +-------+-------+
          |                                |
  +-------v-------+                +-------v-------+
  | Database A    |                | Database B    |
  +---------------+                +---------------+
          |                                |
  +-------v----------------+  +--------v----------------+
  | Message Queue (Events)  |  | Circuit Breaker & Retry |
  +------------------------+  +-------------------------+

Additional components:
+-------------------------+
| Centralized Logging &    |
| Monitoring System        |
+-------------------------+

Components

API Gateway

Nginx, Kong, or AWS API Gateway

Entry point for client requests, routes to appropriate services, handles authentication and rate limiting

Service Registry and Discovery

Consul, Eureka

Keeps track of available service instances for dynamic routing

Circuit Breaker

Hystrix, Resilience4j

Prevents cascading failures by stopping calls to failing services

Message Queue

Kafka, RabbitMQ

Enables asynchronous communication and event-driven architecture for eventual consistency

Centralized Logging and Monitoring

ELK Stack, Prometheus, Grafana

Collects logs and metrics for quick failure detection and troubleshooting

Database per Service

PostgreSQL, MongoDB per service

Ensures data ownership and reduces coupling between services

Deployment Pipeline

Jenkins, GitHub Actions, Spinnaker

Supports automated testing and safe deployments using blue-green or canary strategies

Request Flow

1. Client sends request to API Gateway.

2. API Gateway routes request to appropriate microservice based on URL and service registry.

3. Microservice processes request, reads/writes its own database.

4. If operation requires notifying other services, microservice publishes event to message queue.

5. Other services consume events asynchronously to update their state, ensuring eventual consistency.

6. Circuit breaker monitors service calls; if failures exceed threshold, it trips to prevent further calls.

7. Centralized logging collects logs and metrics from all services for monitoring and alerting.

8. Deployment pipeline enables independent service updates with minimal downtime using canary releases.

Database Schema

Each microservice owns its own database schema. For example: - Service A Database: Table Users(user_id PK, name, email) - Service B Database: Table Orders(order_id PK, user_id FK, product_id, status) Relationships between services are managed via events, not direct DB joins, to reduce coupling.

Scaling Discussion

Bottlenecks

API Gateway becoming a single point of failure or bottleneck under high load

Service-to-service synchronous calls causing cascading failures

Database contention or scaling limits per service

Message queue overload or slow consumers causing event backlog

Deployment errors causing downtime or inconsistent states

Solutions

Use multiple API Gateway instances behind a load balancer for high availability

Implement circuit breakers and bulkheads to isolate failures and prevent cascading

Use database sharding or read replicas to scale databases per service

Scale message queue clusters and optimize consumer throughput; use backpressure mechanisms

Adopt blue-green or canary deployments with automated rollback on failure

Interview Tips

Time: Spend 10 minutes understanding failure causes and clarifying requirements, 20 minutes designing a resilient microservices architecture with failure mitigation, and 15 minutes discussing scaling and operational best practices.

Explain common microservices failure modes like cascading failures and data inconsistency

Describe how patterns like circuit breaker and bulkhead improve resilience

Emphasize importance of asynchronous communication and eventual consistency

Discuss deployment strategies that reduce downtime and risk

Highlight monitoring and alerting as critical for quick failure detection

Practice

1. Which of the following is a key lesson from microservices failures to improve system resilience?

easy

A. Design services to be loosely coupled and handle failures gracefully

B. Combine all services into a single monolith to avoid communication issues

C. Ignore monitoring since failures are rare and unpredictable

D. Avoid retries to prevent additional load on services

Solution

1. Step 1: Understand microservices failure causes

   Failures often happen due to tight coupling and lack of fault tolerance.

2. Step 2: Identify best practice for resilience

   Loose coupling and graceful failure handling improve system stability.

3. Final Answer:

   Design services to be loosely coupled and handle failures gracefully -> Option A

4. Quick Check:

   Loose coupling = resilience [OK]

Hint: Remember: loose coupling prevents cascading failures [OK]

Common Mistakes:

• Thinking monoliths avoid failures
• Ignoring monitoring importance
• Avoiding retries completely

2. Which syntax correctly represents a retry mechanism with a limit in a microservice call?

easy

A. while(true) { callService() }

B. retry(count=-1) { callService() }

C. retry(0) { callService() }

D. retry(count=5) { callService() }

Solution

1. Step 1: Understand retry syntax with limits

   Retries must have a positive count to limit attempts.

2. Step 2: Evaluate options

   retry(count=5) { callService() } uses a positive count (5), valid retry limit; others are infinite or zero retries.

3. Final Answer:

   retry(count=5) { callService() } -> Option D

4. Quick Check:

   Positive retry count = correct syntax [OK]

Hint: Retries need a positive count to avoid infinite loops [OK]

Common Mistakes:

• Using infinite loops for retries
• Setting retry count to zero or negative
• Ignoring retry limits

3. Given this pseudocode for a microservice call with fallback:

result = callService() or fallbackService()

What will be the output if callService() fails but fallbackService() succeeds?

medium

A. An error is thrown and no result is returned

B. The result from callService() is returned despite failure

C. The result from fallbackService() is returned

D. Both results are combined and returned

Solution

1. Step 1: Understand fallback behavior

   If the main service fails, fallback is called to provide a result.

2. Step 2: Analyze given code

   Since callService() fails, fallbackService() result is used.

3. Final Answer:

   The result from fallbackService() is returned -> Option C

4. Quick Check:

   Fallback returns result on failure [OK]

Hint: Fallback runs only if main service fails [OK]

Common Mistakes:

• Assuming error is thrown without fallback
• Thinking main service result returns despite failure
• Believing results combine automatically

4. A microservice call retries 3 times on failure but never succeeds. What is the main issue in this retry design?

medium

A. No fallback mechanism to handle persistent failure

B. Retries cause infinite loops without limits

C. Retries are too few to recover from failure

D. Service calls are synchronous causing delays

Solution

1. Step 1: Analyze retry behavior

   Retries are limited to 3 attempts, so no infinite loop.

2. Step 2: Identify missing resilience feature

   Without fallback, system cannot recover after retries fail.

3. Final Answer:

   No fallback mechanism to handle persistent failure -> Option A

4. Quick Check:

   Retries need fallback for persistent failures [OK]

Hint: Retries alone can't fix persistent failures; add fallback [OK]

Common Mistakes:

• Confusing retry limits with infinite loops
• Assuming more retries always solve failures
• Ignoring fallback importance

5. You design a microservices system where Service A calls Service B, which calls Service C. Service C is unstable and often fails. Which design improves overall system stability best?

hard

A. Make Service A call Service C directly to reduce hops

B. Add retries with limits and fallback in Service B for calls to Service C

C. Remove retries to avoid extra load on Service C

D. Combine Services B and C into one to avoid network calls

Solution

1. Step 1: Identify failure point and impact

   Service C is unstable, causing failures in the chain.

2. Step 2: Apply fault tolerance best practices

   Retries with limits and fallback in Service B isolate failures and improve stability.

3. Step 3: Evaluate other options

   Direct calls or combining services increase coupling or load; removing retries loses resilience.

4. Final Answer:

   Add retries with limits and fallback in Service B for calls to Service C -> Option B

5. Quick Check:

   Retries + fallback near failure = stability [OK]

Hint: Place retries and fallback close to unstable service [OK]

Common Mistakes:

• Increasing coupling by combining services
• Bypassing intermediate services causing tight coupling
• Removing retries losing fault tolerance