Alerting thresholds in HLD - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Alerting Thresholds in Monitoring Systems

Which of the following best describes the purpose of setting alerting thresholds in a system monitoring context?

A To define specific limits that trigger notifications when system metrics exceed or fall below normal ranges

B To automatically fix system errors without human intervention

C To store historical data for long-term analysis without triggering any alerts

D To increase the system's processing speed by adjusting resource allocation dynamically

💡 Hint

Think about why monitoring tools notify engineers about unusual system behavior.

Alerting thresholds are set to notify when metrics go beyond expected values, helping teams respond quickly to issues.

❓ Architecture

intermediate

Designing a Threshold-Based Alerting Component

In a high-level design for an alerting system, which component is primarily responsible for evaluating if a metric breaches its threshold?

A Data Collector

B Data Storage

C Threshold Evaluator

D Notification Dispatcher

💡 Hint

Consider which part checks if values are too high or too low.

The Threshold Evaluator compares incoming metric values against predefined limits to decide if an alert should be triggered.

❓ scaling

advanced

Scaling Alerting Threshold Evaluations for Large Systems

When scaling an alerting system to handle millions of metrics per second, which approach best ensures timely threshold evaluations?

A Centralize all metric evaluations on a single powerful server

B Distribute threshold evaluations across multiple nodes close to data sources

C Disable threshold evaluations to reduce load and rely on manual checks

D Store all metrics first and evaluate thresholds in batch once per day

💡 Hint

Think about reducing latency and avoiding bottlenecks.

Distributing evaluations near data sources reduces latency and balances load, enabling timely alerts at scale.

❓ tradeoff

advanced

Tradeoffs in Setting Alerting Thresholds

What is a common tradeoff when setting very sensitive (low) alerting thresholds in a monitoring system?

A Increased false positives leading to alert fatigue

B Missing critical system failures due to high thresholds

C Reduced system performance due to heavy data storage

D Longer alert delivery times due to batching

💡 Hint

Consider what happens if alerts trigger too often for minor issues.

Very sensitive thresholds cause many alerts, including false alarms, which can overwhelm responders and reduce effectiveness.

❓ estimation

expert

Estimating Alert Volume for Threshold Settings

A system monitors 10,000 metrics, each checked every 10 seconds. If 0.5% of checks breach thresholds on average, estimate how many alerts are generated per hour.

A 3,000 alerts per hour

B 30,000 alerts per hour

C 1,800 alerts per hour

D 18,000 alerts per hour

💡 Hint

Calculate total checks per hour, then apply breach percentage.

Each metric is checked 6 times per minute (60/10), so 10,000 * 6 * 60 = 3,600,000 checks per hour. 0.5% of 3,600,000 is 18,000 alerts.

Practice

1. What is the primary purpose of setting alerting thresholds in system design?

easy

A. To delay notifications until issues become critical

B. To increase system resource usage

C. To reduce the number of system logs generated

D. To detect problems early before they impact users

Solution

1. Step 1: Understand alerting thresholds role

   Alerting thresholds are set to monitor system metrics and detect anomalies early.

2. Step 2: Identify the main goal

   The main goal is to catch issues before they affect users or cause downtime.

3. Final Answer:

   To detect problems early before they impact users -> Option D

4. Quick Check:

   Alerting thresholds = Early problem detection [OK]

Hint: Alerting thresholds catch issues early to protect users [OK]

Common Mistakes:

• Thinking thresholds increase resource use
• Believing thresholds delay notifications
• Confusing thresholds with log management

2. Which of the following is a valid component of an alerting threshold configuration?

easy

A. User interface color scheme

B. Metric name to monitor

C. Database schema version

D. Network cable length

Solution

1. Step 1: Identify components of alerting thresholds

   Alerting thresholds include metric names, threshold values, comparison operators, duration, severity, and notification methods.

2. Step 2: Match valid component

   Metric name is essential to specify what to monitor.

3. Final Answer:

   Metric name to monitor -> Option B

4. Quick Check:

   Alert config includes metric name [OK]

Hint: Alert configs always include metric names [OK]

Common Mistakes:

• Confusing UI settings with alert configs
• Including unrelated hardware details
• Mixing database schema with alert thresholds

3. Consider this alert rule: CPU usage > 80% for 5 minutes triggers WARNING. If CPU usage spikes to 85% for 3 minutes, then drops to 75%, what happens?

medium

A. WARNING alert triggers immediately at 85%

B. Alert triggers after CPU usage drops below 80%

C. No alert is triggered because duration is less than 5 minutes

D. System shuts down due to high CPU

Solution

1. Step 1: Analyze alert condition

   The alert triggers only if CPU usage stays above 80% continuously for 5 minutes.

2. Step 2: Evaluate given scenario

   CPU usage was above 80% for only 3 minutes, which is less than required 5 minutes, so no alert triggers.

3. Final Answer:

   No alert is triggered because duration is less than 5 minutes -> Option C

4. Quick Check:

   Duration condition not met = No alert [OK]

Hint: Duration must be met before alert triggers [OK]

Common Mistakes:

• Ignoring duration and alerting immediately
• Assuming alert triggers after usage drops
• Confusing alert severity with system shutdown

4. You have an alert threshold: Memory usage >= 90% for 10 minutes triggers CRITICAL. The alert never fires even when memory usage is 95% for 15 minutes. What is the most likely issue?

medium

A. The monitoring system is not collecting memory metrics correctly

B. The comparison operator should be > instead of >=

C. The alert duration is set too low

D. The notification channel is misconfigured

Solution

1. Step 1: Check alert condition correctness

   The condition uses >= 90% for 10 minutes, which is valid and should trigger if memory usage is 95% for 15 minutes.

2. Step 2: Identify why alert doesn't fire

   If alert doesn't fire despite condition met, likely the monitoring system is not reporting memory usage correctly.

3. Final Answer:

   The monitoring system is not collecting memory metrics correctly -> Option A

4. Quick Check:

   Metrics missing = Alert won't trigger [OK]

Hint: Check metric collection if alerts don't trigger [OK]

Common Mistakes:

• Changing valid comparison operators unnecessarily
• Assuming duration is too low when it's sufficient
• Blaming notification without checking metrics

5. You design an alerting system with thresholds for CPU, memory, and disk usage. To avoid alert fatigue, which strategy is best?

hard

A. Use multi-level severity with increasing thresholds and durations

B. Disable alerts during business hours

C. Send alerts for every metric spike immediately

D. Set thresholds too low to catch all minor issues

Solution

1. Step 1: Understand alert fatigue

   Alert fatigue happens when too many alerts overwhelm responders, causing important alerts to be missed.

2. Step 2: Choose strategy to reduce noise

   Using multi-level severity with increasing thresholds and durations helps prioritize critical alerts and reduce noise from minor fluctuations.

3. Final Answer:

   Use multi-level severity with increasing thresholds and durations -> Option A

4. Quick Check:

   Multi-level severity reduces alert fatigue [OK]

Hint: Use severity levels and durations to reduce alert noise [OK]

Common Mistakes:

• Setting thresholds too low causing many false alerts
• Alerting on every spike without filtering
• Disabling alerts during critical times