Monitoring and observability in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Monitoring and observability

Which metric matters for Monitoring and Observability and WHY

In monitoring and observability, key metrics include latency, error rate, throughput, and resource usage. These metrics help us understand how well a machine learning model or system is working in real time. For example, latency tells us how fast the model responds, and error rate shows how often it makes mistakes. Observability also involves tracking logs and traces to find hidden problems quickly. These metrics matter because they help keep the system reliable and performant for users.

Confusion matrix or equivalent visualization

While monitoring focuses on system health, for model performance we use a confusion matrix to see prediction quality:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Negative (FN) |
      | False Positive (FP) | True Negative (TN)  |
    

This matrix helps calculate precision, recall, and accuracy, which are important for observability of model quality over time.

Precision vs Recall tradeoff with concrete examples

Monitoring helps us see tradeoffs like precision vs recall. For example, in a spam filter:

• High precision means fewer good emails marked as spam (false alarms).
• High recall means catching most spam emails.

Observability tools track these metrics so we can adjust the model to balance catching spam without losing good emails.

What "good" vs "bad" metric values look like for this use case

Good monitoring metrics show low error rates, stable latency, and consistent throughput. For example:

• Error rate below 1%
• Latency under 100 milliseconds
• Throughput matching expected user load

Bad metrics show spikes in errors, slow responses, or resource overloads, signaling problems needing quick fixes.

Metrics pitfalls

• Accuracy paradox: High accuracy can hide poor performance on rare but important cases.
• Data leakage: Metrics look good because test data leaks into training, misleading monitoring.
• Overfitting indicators: Metrics improve on training data but degrade in real use, showing poor generalization.
• Ignoring latency or resource use: Good accuracy but slow or costly models hurt user experience.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most fraud cases, which is dangerous. High accuracy can be misleading if most transactions are not fraud. Monitoring recall is critical here to catch fraud effectively.

Key Result

Monitoring and observability focus on latency, error rate, and recall to ensure reliable and effective ML system performance.

Practice

1. What is the main purpose of monitoring in a software system?

easy

A. To check if the system is working right now

B. To predict future system failures

C. To change system configurations automatically

D. To write new features for the system

Solution

1. Step 1: Understand monitoring's role

   Monitoring is about checking the current state of the system to see if it is working properly.

2. Step 2: Compare options to definition

   Only To check if the system is working right now matches this purpose. Other options describe different activities like prediction, automation, or development.

3. Final Answer:

   To check if the system is working right now -> Option A

4. Quick Check:

   Monitoring = check current system state [OK]

Hint: Monitoring = check system now, not future or changes [OK]

Common Mistakes:

• Confusing monitoring with observability
• Thinking monitoring predicts future issues
• Assuming monitoring changes system behavior

2. Which of the following is a correct example of a monitoring tool?

easy

A. Visual Studio Code

B. Prometheus

C. Dockerfile

D. GitHub

Solution

1. Step 1: Identify monitoring tools

   Prometheus is a popular open-source monitoring tool used to collect and query metrics.

2. Step 2: Check other options

   GitHub is for code hosting, Dockerfile is for container setup, and Visual Studio Code is a code editor, none are monitoring tools.

3. Final Answer:

   Prometheus -> Option B

4. Quick Check:

   Prometheus = monitoring tool [OK]

Hint: Prometheus is a classic monitoring tool name [OK]

Common Mistakes:

• Confusing code tools with monitoring tools
• Thinking Dockerfile is a monitoring tool
• Mixing development tools with monitoring

3. Given this Prometheus query: up{job="api-server"} == 1, what does it show?

medium

A. The total number of api-server jobs

B. All api-server jobs that are down

C. All api-server jobs that are currently up (running)

D. The CPU usage of api-server jobs

Solution

1. Step 1: Understand the query meaning

   The metric up is 1 when a target is up (running), 0 if down. The filter {job="api-server"} selects only api-server jobs.

2. Step 2: Interpret the comparison

   The query checks where up == 1, so it shows api-server jobs currently running.

3. Final Answer:

   All api-server jobs that are currently up (running) -> Option C

4. Quick Check:

   up == 1 means running targets [OK]

Hint: up == 1 means service is running [OK]

Common Mistakes:

• Thinking up == 1 means down
• Confusing metric with count
• Assuming it shows CPU usage

4. You see this error in your monitoring setup: error parsing query: unexpected token. What is the most likely cause?

medium

A. Server hardware failure

B. Network failure between server and client

C. Monitoring tool is not installed

D. Syntax error in the query expression

Solution

1. Step 1: Analyze the error message

   The message says "error parsing query" and "unexpected token", which means the query syntax is wrong.

2. Step 2: Rule out other causes

   Network failure, missing tool, or hardware failure would cause different errors, not parsing errors.

3. Final Answer:

   Syntax error in the query expression -> Option D

4. Quick Check:

   Parsing error = syntax mistake [OK]

Hint: Parsing errors mean syntax mistakes in queries [OK]

Common Mistakes:

• Assuming network or hardware issues cause parsing errors
• Ignoring the error message details
• Thinking the tool is missing

5. You want to improve observability by adding tracing to your microservices. Which approach best helps you understand why requests fail inside your system?

hard

A. Use distributed tracing to follow requests across services

B. Add more CPU and memory to servers

C. Increase the frequency of monitoring alerts

D. Write more unit tests for each service

Solution

1. Step 1: Understand observability and tracing

   Observability helps explain why things happen. Distributed tracing tracks requests across services to find where failures occur.

2. Step 2: Evaluate options for observability

   Adding resources or alerts or tests does not directly show why requests fail inside the system.

3. Final Answer:

   Use distributed tracing to follow requests across services -> Option A

4. Quick Check:

   Tracing = understand request flow and failures [OK]

Hint: Tracing shows request path and failure reasons [OK]

Common Mistakes:

• Confusing monitoring alerts with observability
• Thinking hardware upgrades improve observability
• Assuming tests replace tracing