Throughput, latency, and availability in HLD - System Design Exercise

Design: Performance Metrics Analysis System

Design focuses on capturing, processing, and reporting throughput, latency, and availability metrics. It excludes detailed application logic or business features.

Functional Requirements

FR1: Measure and report system throughput in requests per second

FR2: Track latency for each request with p50, p95, and p99 percentiles

FR3: Monitor system availability with uptime percentage and downtime alerts

FR4: Provide real-time dashboards for throughput, latency, and availability

FR5: Support alerting when latency or availability thresholds are breached

Non-Functional Requirements

NFR1: Handle up to 100,000 requests per second

NFR2: Latency measurement accuracy within 1 millisecond

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Dashboard updates with less than 5 seconds delay

NFR5: System must be fault tolerant and highly available

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Metrics collection agents or instrumentation

Data ingestion pipeline for metrics

Time-series database for storing metrics

Real-time analytics engine for percentile calculations

Dashboard and alerting system

Design Patterns

Event-driven architecture for metrics ingestion

Sliding window or histogram for latency percentiles

Circuit breaker pattern for availability monitoring

Caching for dashboard performance

Redundancy and failover for high availability

Reference Architecture

  +----------------+       +------------------+       +---------------------+
  | Metrics Source | ----> | Metrics Collector | ----> | Time-Series Database |
  +----------------+       +------------------+       +---------------------+
                                      |                          |
                                      v                          v
                             +----------------+         +----------------+
                             | Analytics Engine|         | Alerting System|
                             +----------------+         +----------------+
                                      |
                                      v
                             +----------------+
                             |   Dashboard    |
                             +----------------+

Components

Metrics Source

Application instrumentation, SDKs

Generate throughput, latency, and availability data from user requests

Metrics Collector

Prometheus, Fluentd, or custom agents

Aggregate and forward metrics data to storage

Time-Series Database

InfluxDB, Prometheus TSDB, or TimescaleDB

Store time-stamped metrics efficiently for querying

Analytics Engine

Apache Flink, Spark Streaming, or custom service

Calculate latency percentiles and throughput in real-time

Alerting System

PagerDuty, Grafana Alerting, or custom alerts

Notify operators when latency or availability thresholds are violated

Dashboard

Grafana, Kibana, or custom web UI

Visualize throughput, latency, and availability metrics in real-time

Request Flow

1. User requests generate metrics data (timestamps, response times, success/failure).

2. Metrics Collector receives data from sources and batches it for efficiency.

3. Data is stored in the Time-Series Database with timestamps for historical analysis.

4. Analytics Engine queries the database continuously to compute throughput and latency percentiles.

5. Alerting System monitors analytics results and triggers alerts on SLA breaches.

6. Dashboard queries analytics and database to display current and historical metrics to users.

Database Schema

Entities: - MetricRecord: {id, timestamp, metric_type (throughput/latency/availability), value, request_id} - Alert: {id, metric_type, threshold, triggered_at, resolved_at, status} Relationships: - MetricRecord stores raw data points linked by timestamp. - Alerts reference metric types and track alert lifecycle.

Scaling Discussion

Bottlenecks

Metrics Collector overwhelmed by high request volume

Time-Series Database storage and query latency under heavy load

Analytics Engine processing delays with large data streams

Alerting System flooding with false positives or missed alerts

Dashboard performance degradation with many concurrent users

Solutions

Use load balancing and sharding for Metrics Collector to distribute ingestion load

Partition Time-Series Database by time and metric type; use compression and downsampling

Scale Analytics Engine horizontally with stream partitioning and parallel processing

Implement adaptive alert thresholds and deduplication to reduce noise

Cache dashboard queries and use CDN for static assets to improve responsiveness

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scale. Use 20 minutes to design components and data flow. Reserve 10 minutes to discuss scaling and trade-offs. Leave 5 minutes for questions.

Explain how throughput, latency, and availability differ and why each matters

Describe how metrics are collected without impacting system performance

Discuss real-time analytics techniques for latency percentiles

Highlight fault tolerance and high availability strategies

Address how to handle scale and maintain low latency in monitoring

Practice

1. Which metric best describes how many requests a system can handle per second?

easy

A. Throughput

B. Latency

C. Availability

D. Scalability

Solution

1. Step 1: Understand throughput meaning

   Throughput measures the amount of work or requests a system can process in a given time.

2. Step 2: Compare with other metrics

   Latency measures response time, availability measures uptime, scalability is about growth capacity.

3. Final Answer:

   Throughput -> Option A

4. Quick Check:

   Throughput = Requests per second [OK]

Hint: Throughput = how much work done per time [OK]

Common Mistakes:

• Confusing latency with throughput
• Mixing availability with throughput
• Thinking scalability equals throughput

2. Which of the following correctly defines latency in system design?

easy

A. The total number of requests processed per second

B. The time taken to respond to a single request

C. The percentage of time the system is operational

D. The ability to add more servers to handle load

Solution

1. Step 1: Define latency

   Latency is the delay or time taken for a system to respond to one request.

2. Step 2: Eliminate other options

   The total number of requests processed per second describes throughput. The ability to add more servers to handle load describes scalability. The percentage of time the system is operational describes availability.

3. Final Answer:

   The time taken to respond to a single request -> Option B

4. Quick Check:

   Latency = Response time [OK]

Hint: Latency = response time per request [OK]

Common Mistakes:

• Confusing latency with throughput
• Thinking availability is latency
• Mixing scalability with latency

3. A system processes 1000 requests in 10 seconds. Each request takes 5 milliseconds to respond. What is the system's throughput and average latency?

medium

A. Throughput: 1000 requests/sec, Latency: 5 ms

B. Throughput: 100 requests/sec, Latency: 50 ms

C. Throughput: 10 requests/sec, Latency: 5 ms

D. Throughput: 100 requests/sec, Latency: 5 ms

Solution

1. Step 1: Calculate throughput

   Throughput = total requests / total time = 1000 / 10 = 100 requests per second.

2. Step 2: Understand latency

   Latency is given as 5 milliseconds per request.

3. Final Answer:

   Throughput: 100 requests/sec, Latency: 5 ms -> Option D

4. Quick Check:

   Throughput = 100, Latency = 5 ms [OK]

Hint: Throughput = total requests ÷ time; latency given directly [OK]

Common Mistakes:

• Dividing latency by number of requests
• Confusing throughput units
• Multiplying latency by total requests

4. A system claims 99.9% availability but is down for 10 minutes in a day. What is the actual availability and is the claim correct?

medium

A. Actual availability is 99.31%, claim is incorrect

B. Actual availability is 99.93%, claim is correct

C. Actual availability is 99.93%, claim is incorrect

D. Actual availability is 99.31%, claim is correct

Solution

1. Step 1: Calculate total minutes in a day

   24 hours x 60 minutes = 1440 minutes per day.

2. Step 2: Calculate availability

   Availability = (Total time - downtime) / Total time = (1440 - 10) / 1440 = 1430 / 1440 ≈ 0.9931 or 99.31%.

3. Step 3: Compare with claim

   Claim is 99.9%, actual is 99.31%, so claim is incorrect.

4. Final Answer:

   Actual availability is 99.31%, claim is incorrect -> Option A

5. Quick Check:

   Availability = 99.31%, Claim wrong [OK]

Hint: Availability = (uptime ÷ total time) x 100% [OK]

Common Mistakes:

• Using hours instead of minutes
• Incorrectly subtracting downtime
• Misreading percentage claim

5. You design a web service that must handle 10,000 requests per second with an average latency under 100 ms and 99.99% availability. Which design choice best balances these requirements?

hard

A. Use a single powerful server with caching to reduce latency

B. Use a database with strong consistency but no replication

C. Distribute load across multiple servers with load balancers and health checks

D. Optimize only latency by increasing CPU speed on one server

Solution

1. Step 1: Analyze throughput and availability needs

   10,000 requests/sec and 99.99% availability require scaling and fault tolerance.

2. Step 2: Evaluate design options

   Using a single powerful server with caching or optimizing CPU speed on one server risks overload and downtime. Using a database with strong consistency but no replication risks availability due to lack of fault tolerance. Distributing load across multiple servers with load balancers and health checks supports scaling and high availability.

3. Final Answer:

   Distribute load across multiple servers with load balancers and health checks -> Option C

4. Quick Check:

   Scaling + availability = load balancer + health checks [OK]

Hint: Scale horizontally with load balancers for throughput and availability [OK]

Common Mistakes:

• Relying on single server for high throughput
• Ignoring replication for availability
• Focusing only on latency without scaling