Redundancy and fault tolerance in HLD - System Design Exercise

Design: Redundancy and Fault Tolerance System

Design focuses on high-level architecture for redundancy and fault tolerance in a web service system. It includes server redundancy, data replication, failure detection, and recovery mechanisms. Out of scope are detailed implementation of business logic and UI design.

Functional Requirements

FR1: Ensure system availability even if some components fail

FR2: Automatically detect failures and recover without manual intervention

FR3: Support continuous operation with minimal downtime

FR4: Provide data replication to avoid data loss

FR5: Allow load distribution to prevent overload on any single component

Non-Functional Requirements

NFR1: System must handle up to 10,000 concurrent users

NFR2: API response latency p99 should be under 300ms

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Recovery time objective (RTO) under 5 minutes

NFR5: Data consistency can be eventual for some components but critical data must be strongly consistent

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Load balancers for distributing traffic

Multiple application servers in different availability zones

Database clusters with replication

Health check and monitoring services

Failover mechanisms and automated recovery scripts

Design Patterns

Replication pattern for data redundancy

Circuit breaker pattern for failure isolation

Heartbeat and health check for failure detection

Leader election for coordinating failover

Retry and exponential backoff for transient errors

Reference Architecture

          +-------------------+          
          |   Load Balancer   |          
          +---------+---------+          
                    |                    
        +-----------+-----------+        
        |                       |        
+-------v-------+       +-------v-------+
| App Server 1  |       | App Server 2  |
+-------+-------+       +-------+-------+
        |                       |        
        +-----------+-----------+        
                    |                    
          +---------v---------+          
          |   Database Cluster |          
          |  (Primary + Replica)|         
          +---------+---------+          
                    |                    
          +---------v---------+          
          |  Monitoring &      |          
          |  Health Checks     |          
          +-------------------+          

Components

Load Balancer

Nginx or AWS ELB

Distributes incoming requests evenly to multiple app servers to avoid overload and provide redundancy

Application Servers

Docker containers on Kubernetes

Run the business logic; multiple instances ensure availability if one fails

Database Cluster

PostgreSQL with streaming replication

Stores data with a primary and one or more replicas for data redundancy and failover

Monitoring & Health Checks

Prometheus and custom health endpoints

Continuously monitor system health and trigger alerts or failover when failures are detected

Failover Mechanism

Patroni or similar leader election tool

Automatically promotes a replica to primary if the primary database fails

Request Flow

1. Client sends request to Load Balancer

2. Load Balancer forwards request to a healthy Application Server

3. Application Server processes request and reads/writes data to Database Cluster

4. Database writes data to primary node and replicates to replicas asynchronously

5. Monitoring system checks health of Application Servers and Database nodes regularly

6. If a failure is detected, failover mechanism promotes a replica to primary

7. Load Balancer stops sending traffic to failed servers and reroutes to healthy ones

Database Schema

Entities: User, Order, Product Relationships: - User 1:N Order (one user can have many orders) - Order N:1 Product (each order relates to one product) Replication: - Primary database node handles writes - Replica nodes asynchronously replicate data for reads and failover

Scaling Discussion

Bottlenecks

Load Balancer can become a single point of failure

Database primary node can become a write bottleneck

Network partitions can cause split-brain scenarios in failover

Monitoring system may not detect failures fast enough

Application servers may exhaust resources under high load

Solutions

Use multiple load balancers with DNS failover or anycast IPs

Implement database sharding or use distributed databases for write scaling

Use consensus protocols (e.g., Raft) to avoid split-brain in leader election

Set aggressive health check intervals and use alerting for quick response

Auto-scale application servers based on CPU/memory usage and request rate

Interview Tips

Time: Spend 10 minutes understanding requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and fault tolerance strategies, 5 minutes for questions and summary.

Explain importance of redundancy to avoid single points of failure

Describe how fault tolerance improves system availability and user experience

Discuss trade-offs between consistency and availability in replication

Highlight automated failure detection and recovery mechanisms

Mention scaling strategies to handle growth and prevent bottlenecks