Throughput, latency, and availability in HLD - System Design Guide

Problem Statement

When a system cannot handle enough requests per second, users experience slow responses or failures. If responses take too long, users get frustrated or abandon the service. If the system is often down or unreachable, users lose trust and stop using it.

Solution

Throughput measures how many requests a system can handle per second, ensuring capacity matches demand. Latency measures the time taken to respond to each request, keeping responses fast and smooth. Availability ensures the system stays up and reachable most of the time, minimizing downtime and failures.

Architecture

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Clients     │──────▶│ Load Balancer │──────▶│   Servers     │
└───────────────┘       └───────────────┘       └───────────────┘
        │                      │                       │
        │                      │                       │
        ▼                      ▼                       ▼
  Throughput: Requests   Latency: Response Time   Availability: Uptime

This diagram shows clients sending requests through a load balancer to servers. Throughput is the number of requests handled, latency is the response time, and availability is the system uptime.

Trade-offs

✓ Pros

→

Measuring throughput helps ensure the system can handle expected user load without bottlenecks.

→

Monitoring latency improves user experience by keeping response times low and predictable.

→

Ensuring high availability builds user trust by minimizing downtime and service interruptions.

✗ Cons

→

Optimizing for high throughput may increase latency if resources are stretched thin.

→

Reducing latency often requires more expensive hardware or caching, increasing costs.

→

Achieving very high availability can add complexity and cost due to redundancy and failover mechanisms.

Use these metrics when designing systems expected to serve thousands or more requests per second, require fast user interactions, and need to be reliable with minimal downtime.

For small-scale systems with very low traffic (under 100 requests per second) or non-critical applications where occasional downtime or delays are acceptable.

Real World Examples

Netflix

Netflix monitors throughput to handle millions of streaming requests, latency to keep video start times low, and availability to ensure users can watch anytime without interruption.

Amazon

Amazon tracks throughput to process thousands of orders per second, latency to keep page loads fast, and availability to maintain 24/7 shopping access.

Uber

Uber measures throughput to handle ride requests, latency to provide quick driver matching, and availability to keep the app operational globally.

Alternatives

Scalability

Scalability focuses on the system's ability to grow capacity, while throughput, latency, and availability are performance metrics to measure system behavior.

Use when: Choose scalability when planning for future growth beyond current throughput or latency limits.

Fault Tolerance

Fault tolerance ensures system continues working despite failures, directly supporting availability but focusing on error handling mechanisms.

Use when: Choose fault tolerance when system must remain operational despite hardware or software failures.

Summary

Throughput, latency, and availability are key metrics to measure system performance and reliability.

Throughput is about how many requests a system can handle, latency is about how fast it responds, and availability is about how often it stays up.

Balancing these metrics helps build systems that are fast, reliable, and scalable for real users.

Practice

1. Which metric best describes how many requests a system can handle per second?

easy

A. Throughput

B. Latency

C. Availability

D. Scalability

Solution

1. Step 1: Understand throughput meaning

   Throughput measures the amount of work or requests a system can process in a given time.

2. Step 2: Compare with other metrics

   Latency measures response time, availability measures uptime, scalability is about growth capacity.

3. Final Answer:

   Throughput -> Option A

4. Quick Check:

   Throughput = Requests per second [OK]

Hint: Throughput = how much work done per time [OK]

Common Mistakes:

• Confusing latency with throughput
• Mixing availability with throughput
• Thinking scalability equals throughput

2. Which of the following correctly defines latency in system design?

easy

A. The total number of requests processed per second

B. The time taken to respond to a single request

C. The percentage of time the system is operational

D. The ability to add more servers to handle load

Solution

1. Step 1: Define latency

   Latency is the delay or time taken for a system to respond to one request.

2. Step 2: Eliminate other options

   The total number of requests processed per second describes throughput. The ability to add more servers to handle load describes scalability. The percentage of time the system is operational describes availability.

3. Final Answer:

   The time taken to respond to a single request -> Option B

4. Quick Check:

   Latency = Response time [OK]

Hint: Latency = response time per request [OK]

Common Mistakes:

• Confusing latency with throughput
• Thinking availability is latency
• Mixing scalability with latency

3. A system processes 1000 requests in 10 seconds. Each request takes 5 milliseconds to respond. What is the system's throughput and average latency?

medium

A. Throughput: 1000 requests/sec, Latency: 5 ms

B. Throughput: 100 requests/sec, Latency: 50 ms

C. Throughput: 10 requests/sec, Latency: 5 ms

D. Throughput: 100 requests/sec, Latency: 5 ms

Solution

1. Step 1: Calculate throughput

   Throughput = total requests / total time = 1000 / 10 = 100 requests per second.

2. Step 2: Understand latency

   Latency is given as 5 milliseconds per request.

3. Final Answer:

   Throughput: 100 requests/sec, Latency: 5 ms -> Option D

4. Quick Check:

   Throughput = 100, Latency = 5 ms [OK]

Hint: Throughput = total requests ÷ time; latency given directly [OK]

Common Mistakes:

• Dividing latency by number of requests
• Confusing throughput units
• Multiplying latency by total requests

4. A system claims 99.9% availability but is down for 10 minutes in a day. What is the actual availability and is the claim correct?

medium

A. Actual availability is 99.31%, claim is incorrect

B. Actual availability is 99.93%, claim is correct

C. Actual availability is 99.93%, claim is incorrect

D. Actual availability is 99.31%, claim is correct

Solution

1. Step 1: Calculate total minutes in a day

   24 hours x 60 minutes = 1440 minutes per day.

2. Step 2: Calculate availability

   Availability = (Total time - downtime) / Total time = (1440 - 10) / 1440 = 1430 / 1440 ≈ 0.9931 or 99.31%.

3. Step 3: Compare with claim

   Claim is 99.9%, actual is 99.31%, so claim is incorrect.

4. Final Answer:

   Actual availability is 99.31%, claim is incorrect -> Option A

5. Quick Check:

   Availability = 99.31%, Claim wrong [OK]

Hint: Availability = (uptime ÷ total time) x 100% [OK]

Common Mistakes:

• Using hours instead of minutes
• Incorrectly subtracting downtime
• Misreading percentage claim

5. You design a web service that must handle 10,000 requests per second with an average latency under 100 ms and 99.99% availability. Which design choice best balances these requirements?

hard

A. Use a single powerful server with caching to reduce latency

B. Use a database with strong consistency but no replication

C. Distribute load across multiple servers with load balancers and health checks

D. Optimize only latency by increasing CPU speed on one server

Solution

1. Step 1: Analyze throughput and availability needs

   10,000 requests/sec and 99.99% availability require scaling and fault tolerance.

2. Step 2: Evaluate design options

   Using a single powerful server with caching or optimizing CPU speed on one server risks overload and downtime. Using a database with strong consistency but no replication risks availability due to lack of fault tolerance. Distributing load across multiple servers with load balancers and health checks supports scaling and high availability.

3. Final Answer:

   Distribute load across multiple servers with load balancers and health checks -> Option C

4. Quick Check:

   Scaling + availability = load balancer + health checks [OK]

Hint: Scale horizontally with load balancers for throughput and availability [OK]

Common Mistakes:

• Relying on single server for high throughput
• Ignoring replication for availability
• Focusing only on latency without scaling