Monitoring and alerting in production in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

What is the output of this Prometheus query?

Given the Prometheus query rate(http_requests_total[5m]), what does it return?

LangChain

rate(http_requests_total[5m])

A The current number of HTTP requests being processed.

B The total number of HTTP requests since the server started.

C The maximum number of HTTP requests in any 5-minute window.

D The average number of HTTP requests per second over the last 5 minutes.

💡 Hint

Think about what the 'rate' function calculates over a time range.

The rate() function calculates the per-second average rate of increase of a counter over the specified time window, here 5 minutes.

🧠 Conceptual

intermediate

Which alerting condition triggers a PagerDuty notification?

You have an alert rule: IF cpu_usage > 90% FOR 10m THEN alert. Which option best describes when PagerDuty will be notified?

A PagerDuty is notified immediately when CPU usage exceeds 90%.

B PagerDuty is notified only if CPU usage stays above 90% continuously for 10 minutes.

C PagerDuty is notified if CPU usage exceeds 90% at any point during 10 minutes, even briefly.

D PagerDuty is notified after CPU usage drops below 90% following a spike.

💡 Hint

Consider the meaning of the 'FOR 10m' clause in alert rules.

The 'FOR 10m' means the condition must be true continuously for 10 minutes before triggering the alert and notification.

❓ Troubleshoot

advanced

Why does this Grafana alert never fire?

You created a Grafana alert with this query: memory_usage[5m] > 80. The alert never fires even when memory usage is high. What is the likely cause?

LangChain

memory_usage[5m] > 80

A The <code>memory_usage[5m]</code> selector returns a range vector, which cannot be directly compared to a scalar.

B The alert condition compares a vector to a scalar without proper aggregation.

C The query uses a wrong function; it should be <code>avg(memory_usage) > 80</code>.

D The threshold 80 is too high; memory usage never reaches that value.

💡 Hint

Check the data types returned by Prometheus functions and how comparisons work.

memory_usage[5m] returns a range vector, but alert conditions require instant vectors for comparison to scalars.

🔀 Workflow

advanced

Order the steps to set up a basic alerting pipeline

Arrange these steps in the correct order to set up monitoring and alerting for a web service.

A 1,2,4,3

B 4,1,2,3

C 1,4,2,3

D 4,2,1,3

💡 Hint

Think about what must be ready before alert rules and notifications.

First, metrics must be exposed by the service (1), then the monitoring system must collect them (4), then alert rules are defined (2), and finally notifications are set up (3).

✅ Best Practice

expert

Which practice improves alert reliability and reduces noise?

You notice many alerts firing for brief spikes in CPU usage. Which practice best reduces false alerts without missing real issues?

A Use alert 'for' duration to require condition to persist before firing.

B Set alert thresholds very high to avoid triggering on spikes.

C Disable alerts during peak traffic hours to reduce noise.

D Send alerts to multiple channels to ensure visibility.

💡 Hint

Consider how to avoid alerts on short-lived spikes.

Using an alert 'for' duration means the condition must be true continuously for a set time, reducing noise from brief spikes.

Practice

1. What is the main purpose of monitoring in a production environment?

easy

A. To send immediate messages when problems happen

B. To backup data regularly

C. To deploy new features automatically

D. To watch the app's health and performance continuously

Solution

1. Step 1: Understand monitoring role

   Monitoring means watching the app's health and performance over time.

2. Step 2: Differentiate from alerting

   Alerting is about sending messages when issues occur, not continuous watching.

3. Final Answer:

   To watch the app's health and performance continuously -> Option D

4. Quick Check:

   Monitoring = watch app health [OK]

Hint: Monitoring means watching, alerting means notifying [OK]

Common Mistakes:

• Confusing monitoring with alerting
• Thinking monitoring deploys features
• Mixing monitoring with backups

2. Which of the following is the correct way to define an alert condition in a monitoring tool?

easy

A. alert every 10 minutes regardless of CPU usage

B. alert when CPU usage > 80% for 5 minutes

C. alert when CPU usage equals 50%

D. alert if CPU usage less than 80%

Solution

1. Step 1: Identify proper alert condition

   An alert should trigger when a metric exceeds a threshold for a time period, e.g., CPU usage > 80% for 5 minutes.

2. Step 2: Eliminate incorrect options

   Alerts on less than threshold or exact equals are less useful; alerting regardless of usage is noisy.

3. Final Answer:

   alert when CPU usage > 80% for 5 minutes -> Option B

4. Quick Check:

   Alert condition = threshold + duration [OK]

Hint: Alert triggers on threshold breach over time [OK]

Common Mistakes:

• Setting alerts on exact equals
• Alerting on low usage instead of high
• Alerting without condition or duration

3. Given this alert rule snippet:

if error_rate > 5% for 10 minutes then send alert

What happens if error_rate spikes to 6% for 8 minutes and then drops to 4%?

medium

A. No alert is sent because the condition duration is not met

B. An alert is sent immediately when error_rate hits 6%

C. An alert is sent after 8 minutes

D. An alert is sent after error_rate drops below 5%

Solution

1. Step 1: Understand alert duration condition

   The alert triggers only if error_rate > 5% continuously for 10 minutes.

2. Step 2: Analyze given scenario

   Error rate was above 5% for 8 minutes, which is less than 10 minutes, so alert does not trigger.

3. Final Answer:

   No alert is sent because the condition duration is not met -> Option A

4. Quick Check:

   Duration condition unmet = no alert [OK]

Hint: Alert needs full duration breach, not just spike [OK]

Common Mistakes:

• Assuming alert triggers immediately on threshold breach
• Ignoring duration requirement
• Thinking alert triggers after drop below threshold

4. You set an alert to notify your team when memory usage exceeds 90%, but no alerts are received even though memory usage is high. What is the most likely cause?

medium

A. Notification channel is not configured correctly

B. Memory usage metric is not collected

C. Alert condition threshold is set too low

D. Alert duration is set to zero

Solution

1. Step 1: Check alert condition and metric

   Memory usage is high, so condition threshold is likely correct and metric is collected.

2. Step 2: Verify notification setup

   If no alerts are received, the notification channel (email, Slack, etc.) may be misconfigured or missing.

3. Final Answer:

   Notification channel is not configured correctly -> Option A

4. Quick Check:

   No alerts + correct condition = notification issue [OK]

Hint: Check notification setup if alerts not received [OK]

Common Mistakes:

• Assuming threshold is always wrong
• Ignoring notification channel setup
• Thinking metric collection is always faulty

5. You want to monitor a LangChain app's response time and alert the team if the average response time exceeds 2 seconds over 15 minutes. Which approach best achieves this?

hard

A. Monitor only error rates and ignore response time

B. Send an alert every time a single response takes longer than 2 seconds

C. Set up a monitoring metric for response time and alert if average > 2s for 15 minutes

D. Alert if any response time is exactly 2 seconds

Solution

1. Step 1: Define monitoring metric and alert condition

   Track average response time metric over 15 minutes to smooth out spikes.

2. Step 2: Set alert on average exceeding threshold

   Alert triggers only if average response time is above 2 seconds for the full 15 minutes.

3. Final Answer:

   Set up a monitoring metric for response time and alert if average > 2s for 15 minutes -> Option C

4. Quick Check:

   Average metric + duration alert = best practice [OK]

Hint: Alert on average over time, not single spikes [OK]

Common Mistakes:

• Alerting on single slow response
• Ignoring response time monitoring
• Alerting on exact value matches