Redundancy and fault tolerance in HLD - Scalability & System Analysis

Scalability Analysis - Redundancy and fault tolerance

Growth Table: Redundancy and Fault Tolerance at Different Scales

Scale	System Setup	Redundancy Level	Fault Tolerance Features	Monitoring & Recovery
100 users	Single server with backup	Basic: one backup server or snapshot	Simple failover, manual recovery	Basic alerts, manual checks
10,000 users	Multiple servers with load balancer	Active-passive redundancy	Automatic failover, health checks	Automated monitoring, alerting
1,000,000 users	Distributed servers across regions	Active-active redundancy, data replication	Automatic failover, self-healing, data consistency checks	Advanced monitoring, auto-scaling, incident response
100,000,000 users	Global multi-region clusters with microservices	Multi-level redundancy (network, compute, storage)	Geo-redundancy, disaster recovery, chaos engineering	AI-driven monitoring, predictive failure detection

First Bottleneck

At small scale, the first bottleneck is often the single point of failure in the server or storage. Without redundancy, any hardware or software failure causes downtime.

As users grow, network and data replication delays become bottlenecks. Ensuring data consistency across redundant systems can slow down operations.

At very large scale, complexity in managing failover and recovery across regions can cause delays and partial outages.

Scaling Solutions

• Horizontal scaling: Add more servers to distribute load and provide redundancy.
• Active-active redundancy: Run multiple instances simultaneously to avoid downtime.
• Data replication: Use synchronous or asynchronous replication to keep copies of data.
• Load balancers: Detect failures and route traffic to healthy servers.
• Health checks and monitoring: Automatically detect failures and trigger failover.
• Disaster recovery plans: Backup data and systems in different geographic locations.
• Chaos engineering: Test system resilience by simulating failures.

Back-of-Envelope Cost Analysis

Requests per second (RPS):

• 100 users: ~10-50 RPS
• 10,000 users: ~1,000-5,000 RPS
• 1,000,000 users: ~100,000-500,000 RPS
• 100,000,000 users: ~10,000,000+ RPS

Storage needs increase with data replication and backups. For example, 1 TB primary data may require 2-3 TB total with redundancy.

Network bandwidth must support replication traffic and failover synchronization, which grows with scale.

Interview Tip

When discussing redundancy and fault tolerance, start by identifying single points of failure. Then explain how to add redundancy at each layer: compute, storage, network.

Describe trade-offs between synchronous and asynchronous replication, and how failover is detected and handled.

Use real-world analogies like backup generators or multiple cashiers to explain concepts simply.

Self Check Question

Your database handles 1000 QPS. Traffic grows 10x. What do you do first?

Answer: Add read replicas and implement caching to reduce load on the primary database before scaling vertically or sharding.

Key Result

Redundancy and fault tolerance start by removing single points of failure and adding backups. As scale grows, data replication, automatic failover, and multi-region setups become critical to maintain availability and performance.

Practice

1. What is the main purpose of redundancy in system design?

easy

A. To simplify the system architecture

B. To increase the speed of data processing

C. To have backup components to avoid single points of failure

D. To reduce the cost of hardware

Solution

1. Step 1: Understand redundancy concept

   Redundancy means having extra components or backups so if one fails, the system still works.

2. Step 2: Identify the main goal of redundancy

   The goal is to avoid single points of failure by having backups ready.

3. Final Answer:

   To have backup components to avoid single points of failure -> Option C

4. Quick Check:

   Redundancy = Backup components [OK]

Hint: Redundancy means backups to prevent failure [OK]

Common Mistakes:

• Confusing redundancy with performance improvement
• Thinking redundancy reduces cost
• Assuming redundancy simplifies architecture

2. Which of the following is a correct example of a fault-tolerant system component?

easy

A. A load balancer distributing traffic to multiple servers

B. A single server with no backup

C. A database without replication

D. A system that shuts down on any error

Solution

1. Step 1: Identify fault tolerance features

   Fault tolerance means the system can keep working even if some parts fail.

2. Step 2: Evaluate options for fault tolerance

   A load balancer sends traffic to multiple servers, so if one fails, others handle requests.

3. Final Answer:

   A load balancer distributing traffic to multiple servers -> Option A

4. Quick Check:

   Fault tolerance = multiple servers with load balancer [OK]

Hint: Fault tolerance needs multiple working parts, not single points [OK]

Common Mistakes:

• Choosing single server as fault tolerant
• Ignoring replication or backups
• Confusing fault tolerance with error shutdown

3. Consider a system with two identical servers running in active-active mode for fault tolerance. If one server fails, what happens to the incoming requests?

medium

A. The system shuts down to prevent errors

B. All requests are lost until the failed server is fixed

C. Requests are queued indefinitely with no processing

D. Requests are automatically handled by the remaining active server

Solution

1. Step 1: Understand active-active mode

   In active-active, both servers handle requests simultaneously to share load and provide backup.

2. Step 2: Analyze failure impact

   If one server fails, the other continues handling all requests without interruption.

3. Final Answer:

   Requests are automatically handled by the remaining active server -> Option D

4. Quick Check:

   Active-active means failover to active server [OK]

Hint: Active-active means all servers handle requests; failure shifts load [OK]

Common Mistakes:

• Thinking requests are lost on failure
• Assuming system queues requests indefinitely
• Believing system shuts down on failure

4. A system uses a single database without replication. It crashes unexpectedly. What is the main fault tolerance issue here?

medium

A. Load balancer misconfiguration

B. No backup database to take over on failure

C. Too many replicas causing conflicts

D. Excessive redundancy causing delays

Solution

1. Step 1: Identify fault tolerance problem

   Fault tolerance requires backups or replicas to continue working if one fails.

2. Step 2: Analyze the system setup

   Single database without replication means no backup exists to handle failure.

3. Final Answer:

   No backup database to take over on failure -> Option B

4. Quick Check:

   No replication = no fault tolerance backup [OK]

Hint: No replication means no fault tolerance backup [OK]

Common Mistakes:

• Confusing replication with too many replicas
• Blaming load balancer for database crash
• Thinking redundancy causes delays here

5. You are designing a fault-tolerant web service that must remain available even if one data center fails. Which design choice best supports this requirement?

hard

A. Deploy the service in two geographically separate data centers with data replication

B. Deploy the service in one data center with multiple servers

C. Use a single powerful server with backup power supply

D. Deploy the service in one data center and rely on daily backups

Solution

1. Step 1: Understand availability requirement

   The service must stay available even if one entire data center fails.

2. Step 2: Evaluate design options for fault tolerance

   Deploying in two separate data centers with data replication ensures if one fails, the other serves requests.

3. Step 3: Compare other options

   Single data center or single server cannot handle full data center failure; daily backups do not provide immediate availability.

4. Final Answer:

   Deploy the service in two geographically separate data centers with data replication -> Option A

5. Quick Check:

   Geo-redundancy + replication = high availability [OK]

Hint: Use multiple data centers with replication for fault tolerance [OK]

Common Mistakes:

• Relying on single data center for fault tolerance
• Confusing backup power with data center failure protection
• Thinking daily backups ensure immediate availability