Why monitoring detects issues before users do in HLD - Design It to Understand It

Design: Monitoring System for Early Issue Detection

Design the monitoring system architecture focusing on early detection mechanisms. Out of scope: detailed alerting rules or user notification channels.

Functional Requirements

FR1: Continuously track system health metrics like CPU, memory, and response times

FR2: Detect anomalies or failures before they impact users

FR3: Send alerts to engineers when issues arise

FR4: Provide dashboards for real-time visibility

FR5: Support multiple system components and services

Non-Functional Requirements

NFR1: Must handle data from thousands of servers and services

NFR2: Alert latency under 1 minute from issue detection

NFR3: High availability with 99.9% uptime

NFR4: Minimal performance impact on monitored systems

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Metric collectors/agents on servers

Centralized time-series database

Alerting engine with threshold and anomaly detection

Dashboard and visualization service

Notification system for alerts

Design Patterns

Push vs pull metric collection

Event-driven alerting

Circuit breaker pattern for fault tolerance

Sampling and aggregation to reduce data volume

Health check and heartbeat mechanisms

Reference Architecture

Metric Collectors

↓

Central Monitoring Server

↓

Time-Series DB

↓

Notification Service

↓

Dashboard Service

Components

Metric Collectors

Prometheus Node Exporter or custom agents

Gather system metrics and send to central server

Central Monitoring Server

Prometheus or similar

Aggregate metrics and store in time-series database

Time-Series Database

Prometheus TSDB, InfluxDB

Efficient storage and querying of time-stamped metrics

Alert Engine

Prometheus Alertmanager or custom rules engine

Evaluate metrics against thresholds and detect anomalies

Notification Service

Email, SMS, PagerDuty integration

Send alerts to engineers promptly

Dashboard Service

Grafana or custom UI

Visualize metrics and system health in real-time

Request Flow

1. 1. Metric collectors on user systems gather CPU, memory, response time, and error rate metrics.

2. 2. Collectors push or expose metrics to the central monitoring server at regular intervals.

3. 3. Central server stores metrics in a time-series database for efficient querying.

4. 4. Alert engine continuously evaluates metrics against predefined thresholds or anomaly detection algorithms.

5. 5. When an issue is detected, alert engine triggers notifications via the notification service.

6. 6. Engineers receive alerts before users experience problems.

7. 7. Dashboard service provides real-time visualization for monitoring and troubleshooting.

Database Schema

Entities: - Metric: id, name, timestamp, value, source_system - AlertRule: id, metric_name, threshold, condition, severity - Alert: id, alert_rule_id, timestamp, status, description Relationships: - Metric data linked to source systems - Alert linked to AlertRule that triggered it

Scaling Discussion

Bottlenecks

High volume of metrics causing storage and query performance issues

Alert engine overwhelmed with frequent alerts causing noise

Network overhead from metric collection agents

Dashboard latency with large data sets

Solutions

Use metric aggregation and downsampling to reduce data volume

Implement alert deduplication and rate limiting to reduce noise

Use pull-based metric collection to control load

Shard time-series database and use caching for dashboards

Interview Tips

Time: Spend 10 minutes understanding requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain why early detection helps prevent user impact

Discuss metric types and collection methods

Describe alerting mechanisms and thresholds

Highlight importance of low latency and high availability

Address scaling challenges and mitigation strategies

Practice

1. Why does monitoring detect issues before users notice them?

easy

A. Because it continuously checks system health and alerts early

B. Because users report problems faster than monitoring

C. Because monitoring waits for user complaints to start

D. Because it only checks the system once a day

Solution

1. Step 1: Understand monitoring's role

   Monitoring tools run all the time to watch system status without breaks.

2. Step 2: Compare monitoring to user experience

   Users only notice problems after they affect them, but monitoring alerts before impact.

3. Final Answer:

   Because it continuously checks system health and alerts early -> Option A

4. Quick Check:

   Continuous checks = Early alerts [OK]

Hint: Monitoring runs nonstop to catch issues early [OK]

Common Mistakes:

• Thinking users report issues faster
• Believing monitoring checks only once
• Assuming monitoring waits for complaints

2. Which command syntax correctly checks system health using a monitoring tool?

easy

A. check_status monitor

B. monitor check status --

C. monitor --check-status

D. status --monitor check

Solution

1. Step 1: Identify correct command structure

   Commands usually start with the tool name followed by options or flags.

2. Step 2: Match syntax to common patterns

   monitor --check-status uses the correct order: tool name 'monitor' then flag '--check-status'.

3. Final Answer:

   monitor --check-status -> Option C

4. Quick Check:

   Tool name first, then flags [OK]

Hint: Tool name comes first, then options [OK]

Common Mistakes:

• Putting flags before tool name
• Using spaces incorrectly
• Mixing order of command parts

3. Given this monitoring output:

CPU Usage: 85%
Memory Usage: 70%
Disk Space: 95%

What should the monitoring system do next?

medium

A. Ignore because all values are normal

B. Alert because disk space is critically high

C. Restart the system immediately

D. Shut down monitoring to save resources

Solution

1. Step 1: Analyze resource usage values

   CPU and memory are high but not critical; disk space at 95% is very high.

2. Step 2: Determine appropriate monitoring action

   High disk usage should trigger an alert to prevent failures.

3. Final Answer:

   Alert because disk space is critically high -> Option B

4. Quick Check:

   Critical disk usage = Alert [OK]

Hint: High disk usage means alert, not ignore [OK]

Common Mistakes:

• Ignoring high disk usage
• Restarting system without cause
• Stopping monitoring unexpectedly

4. A monitoring alert is not sent even though CPU usage is 99%. What is the likely problem?

medium

A. Alert threshold is set too high

B. Monitoring tool is not installed

C. CPU usage is not monitored by default

D. Users disabled alerts manually

Solution

1. Step 1: Check alert configuration

   If CPU usage is very high but no alert, threshold might be set above 99%, so no alert triggers.

2. Step 2: Rule out other causes

   Monitoring tool must be installed to get data; CPU is usually monitored; user disabling alerts is less common.

3. Final Answer:

   Alert threshold is set too high -> Option A

4. Quick Check:

   High threshold = no alert [OK]

Hint: Check alert thresholds if no alerts on high usage [OK]

Common Mistakes:

• Assuming tool is missing without checking
• Ignoring alert configuration
• Blaming users without evidence

5. You want to ensure your monitoring system detects issues before users do. Which combined approach is best?

hard

A. Disable alerts to avoid false alarms

B. Check system health only during business hours

C. Rely on user feedback to find problems

D. Set low alert thresholds and run monitoring continuously

Solution

1. Step 1: Understand monitoring frequency

   Continuous monitoring ensures no gaps in watching system health.

2. Step 2: Set alert thresholds carefully

   Low thresholds catch problems early before they affect users.

3. Step 3: Evaluate other options

   Checking only business hours misses issues; relying on users delays detection; disabling alerts hides problems.

4. Final Answer:

   Set low alert thresholds and run monitoring continuously -> Option D

5. Quick Check:

   Continuous + low thresholds = early detection [OK]

Hint: Combine continuous monitoring with low alert levels [OK]

Common Mistakes:

• Monitoring only part-time
• Ignoring alerts
• Waiting for user reports