Redundancy and fault tolerance in HLD - System Design Exercise

Design: Redundancy and Fault Tolerance System

Design focuses on high-level architecture for redundancy and fault tolerance in a web service system. It includes server redundancy, data replication, failure detection, and recovery mechanisms. Out of scope are detailed implementation of business logic and UI design.

Functional Requirements

FR1: Ensure system availability even if some components fail

FR2: Automatically detect failures and recover without manual intervention

FR3: Support continuous operation with minimal downtime

FR4: Provide data replication to avoid data loss

FR5: Allow load distribution to prevent overload on any single component

Non-Functional Requirements

NFR1: System must handle up to 10,000 concurrent users

NFR2: API response latency p99 should be under 300ms

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Recovery time objective (RTO) under 5 minutes

NFR5: Data consistency can be eventual for some components but critical data must be strongly consistent

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Load balancers for distributing traffic

Multiple application servers in different availability zones

Database clusters with replication

Health check and monitoring services

Failover mechanisms and automated recovery scripts

Design Patterns

Replication pattern for data redundancy

Circuit breaker pattern for failure isolation

Heartbeat and health check for failure detection

Leader election for coordinating failover

Retry and exponential backoff for transient errors

Reference Architecture

          +-------------------+          
          |   Load Balancer   |          
          +---------+---------+          
                    |                    
        +-----------+-----------+        
        |                       |        
+-------v-------+       +-------v-------+
| App Server 1  |       | App Server 2  |
+-------+-------+       +-------+-------+
        |                       |        
        +-----------+-----------+        
                    |                    
          +---------v---------+          
          |   Database Cluster |          
          |  (Primary + Replica)|         
          +---------+---------+          
                    |                    
          +---------v---------+          
          |  Monitoring &      |          
          |  Health Checks     |          
          +-------------------+          

Components

Load Balancer

Nginx or AWS ELB

Distributes incoming requests evenly to multiple app servers to avoid overload and provide redundancy

Application Servers

Docker containers on Kubernetes

Run the business logic; multiple instances ensure availability if one fails

Database Cluster

PostgreSQL with streaming replication

Stores data with a primary and one or more replicas for data redundancy and failover

Monitoring & Health Checks

Prometheus and custom health endpoints

Continuously monitor system health and trigger alerts or failover when failures are detected

Failover Mechanism

Patroni or similar leader election tool

Automatically promotes a replica to primary if the primary database fails

Request Flow

1. Client sends request to Load Balancer

2. Load Balancer forwards request to a healthy Application Server

3. Application Server processes request and reads/writes data to Database Cluster

4. Database writes data to primary node and replicates to replicas asynchronously

5. Monitoring system checks health of Application Servers and Database nodes regularly

6. If a failure is detected, failover mechanism promotes a replica to primary

7. Load Balancer stops sending traffic to failed servers and reroutes to healthy ones

Database Schema

Entities: User, Order, Product Relationships: - User 1:N Order (one user can have many orders) - Order N:1 Product (each order relates to one product) Replication: - Primary database node handles writes - Replica nodes asynchronously replicate data for reads and failover

Scaling Discussion

Bottlenecks

Load Balancer can become a single point of failure

Database primary node can become a write bottleneck

Network partitions can cause split-brain scenarios in failover

Monitoring system may not detect failures fast enough

Application servers may exhaust resources under high load

Solutions

Use multiple load balancers with DNS failover or anycast IPs

Implement database sharding or use distributed databases for write scaling

Use consensus protocols (e.g., Raft) to avoid split-brain in leader election

Set aggressive health check intervals and use alerting for quick response

Auto-scale application servers based on CPU/memory usage and request rate

Interview Tips

Time: Spend 10 minutes understanding requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and fault tolerance strategies, 5 minutes for questions and summary.

Explain importance of redundancy to avoid single points of failure

Describe how fault tolerance improves system availability and user experience

Discuss trade-offs between consistency and availability in replication

Highlight automated failure detection and recovery mechanisms

Mention scaling strategies to handle growth and prevent bottlenecks

Practice

1. What is the main purpose of redundancy in system design?

easy

A. To simplify the system architecture

B. To increase the speed of data processing

C. To have backup components to avoid single points of failure

D. To reduce the cost of hardware

Solution

1. Step 1: Understand redundancy concept

   Redundancy means having extra components or backups so if one fails, the system still works.

2. Step 2: Identify the main goal of redundancy

   The goal is to avoid single points of failure by having backups ready.

3. Final Answer:

   To have backup components to avoid single points of failure -> Option C

4. Quick Check:

   Redundancy = Backup components [OK]

Hint: Redundancy means backups to prevent failure [OK]

Common Mistakes:

• Confusing redundancy with performance improvement
• Thinking redundancy reduces cost
• Assuming redundancy simplifies architecture

2. Which of the following is a correct example of a fault-tolerant system component?

easy

A. A load balancer distributing traffic to multiple servers

B. A single server with no backup

C. A database without replication

D. A system that shuts down on any error

Solution

1. Step 1: Identify fault tolerance features

   Fault tolerance means the system can keep working even if some parts fail.

2. Step 2: Evaluate options for fault tolerance

   A load balancer sends traffic to multiple servers, so if one fails, others handle requests.

3. Final Answer:

   A load balancer distributing traffic to multiple servers -> Option A

4. Quick Check:

   Fault tolerance = multiple servers with load balancer [OK]

Hint: Fault tolerance needs multiple working parts, not single points [OK]

Common Mistakes:

• Choosing single server as fault tolerant
• Ignoring replication or backups
• Confusing fault tolerance with error shutdown

3. Consider a system with two identical servers running in active-active mode for fault tolerance. If one server fails, what happens to the incoming requests?

medium

A. The system shuts down to prevent errors

B. All requests are lost until the failed server is fixed

C. Requests are queued indefinitely with no processing

D. Requests are automatically handled by the remaining active server

Solution

1. Step 1: Understand active-active mode

   In active-active, both servers handle requests simultaneously to share load and provide backup.

2. Step 2: Analyze failure impact

   If one server fails, the other continues handling all requests without interruption.

3. Final Answer:

   Requests are automatically handled by the remaining active server -> Option D

4. Quick Check:

   Active-active means failover to active server [OK]

Hint: Active-active means all servers handle requests; failure shifts load [OK]

Common Mistakes:

• Thinking requests are lost on failure
• Assuming system queues requests indefinitely
• Believing system shuts down on failure

4. A system uses a single database without replication. It crashes unexpectedly. What is the main fault tolerance issue here?

medium

A. Load balancer misconfiguration

B. No backup database to take over on failure

C. Too many replicas causing conflicts

D. Excessive redundancy causing delays

Solution

1. Step 1: Identify fault tolerance problem

   Fault tolerance requires backups or replicas to continue working if one fails.

2. Step 2: Analyze the system setup

   Single database without replication means no backup exists to handle failure.

3. Final Answer:

   No backup database to take over on failure -> Option B

4. Quick Check:

   No replication = no fault tolerance backup [OK]

Hint: No replication means no fault tolerance backup [OK]

Common Mistakes:

• Confusing replication with too many replicas
• Blaming load balancer for database crash
• Thinking redundancy causes delays here

5. You are designing a fault-tolerant web service that must remain available even if one data center fails. Which design choice best supports this requirement?

hard

A. Deploy the service in two geographically separate data centers with data replication

B. Deploy the service in one data center with multiple servers

C. Use a single powerful server with backup power supply

D. Deploy the service in one data center and rely on daily backups

Solution

1. Step 1: Understand availability requirement

   The service must stay available even if one entire data center fails.

2. Step 2: Evaluate design options for fault tolerance

   Deploying in two separate data centers with data replication ensures if one fails, the other serves requests.

3. Step 3: Compare other options

   Single data center or single server cannot handle full data center failure; daily backups do not provide immediate availability.

4. Final Answer:

   Deploy the service in two geographically separate data centers with data replication -> Option A

5. Quick Check:

   Geo-redundancy + replication = high availability [OK]

Hint: Use multiple data centers with replication for fault tolerance [OK]

Common Mistakes:

• Relying on single data center for fault tolerance
• Confusing backup power with data center failure protection
• Thinking daily backups ensure immediate availability