Automated evaluation metrics in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Automated evaluation metrics

Which metric matters for Automated evaluation metrics and WHY

Automated evaluation metrics help us quickly check how well a model is doing without needing humans every time. The right metric depends on the task:

• Accuracy measures overall correct predictions, good for balanced classes.
• Precision tells us how many predicted positives are actually correct, important when false alarms are costly.
• Recall shows how many real positives we found, key when missing positives is bad.
• F1 Score balances precision and recall, useful when both matter.
• AUC (Area Under Curve) measures how well the model separates classes, useful for ranking tasks.

Choosing the right metric helps us understand if the model fits our real-world needs.

Confusion matrix example

      |                    | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|--------------------|
      | Actual Positive     | True Positive (TP)  | False Negative (FN) |
      | Actual Negative     | False Positive (FP) | True Negative (TN)  |

      Example numbers:
      TP = 70, FP = 10, TN = 900, FN = 20

      Total samples = 70 + 10 + 900 + 20 = 1000
    

From this, we calculate:

• Precision = 70 / (70 + 10) = 0.875
• Recall = 70 / (70 + 20) = 0.778
• Accuracy = (70 + 900) / 1000 = 0.97
• F1 Score = 2 * (0.875 * 0.778) / (0.875 + 0.778) ≈ 0.823

Precision vs Recall tradeoff with examples

Precision and recall often pull in opposite directions:

• High Precision, Low Recall: The model is careful and only predicts positive when very sure. Good for spam filters so normal emails aren't marked spam.
• High Recall, Low Precision: The model catches almost all positives but may include many false alarms. Good for cancer detection so no cancer case is missed.

Automated metrics help us find the right balance based on what matters more in our problem.

What "good" vs "bad" metric values look like

For automated evaluation metrics, here is what to expect:

• Good: High precision and recall (above 0.8) means the model predicts well and finds most positives.
• Bad: Low precision (<0.5) means many false alarms; low recall (<0.5) means many missed positives.
• Accuracy: High accuracy (>0.9) is good if classes are balanced, but can be misleading if data is skewed.
• F1 Score: A balanced score above 0.7 is usually acceptable.

Common pitfalls in automated evaluation metrics

• Accuracy paradox: High accuracy can hide poor performance if classes are imbalanced.
• Data leakage: When test data leaks into training, metrics look unrealistically good.
• Overfitting indicators: Very high training metrics but low test metrics mean the model memorizes instead of learning.
• Ignoring context: Using the wrong metric for the problem can mislead decisions.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud is rare, so the model mostly predicts non-fraud correctly but fails to catch fraud.

Key Result

Automated evaluation metrics like precision, recall, and F1 score provide clear, task-relevant insights to judge model quality beyond simple accuracy.

Practice

1. Which automated evaluation metric is commonly used to measure the accuracy of classification models?

easy

A. Perplexity

B. Mean Squared Error

C. BLEU Score

D. Accuracy

Solution

1. Step 1: Understand classification metrics

   Classification models predict categories, so metrics like Accuracy measure correct predictions over total predictions.

2. Step 2: Match metric to task

   Mean Squared Error is for regression, BLEU and Perplexity are for language tasks, so Accuracy fits classification best.

3. Final Answer:

   Accuracy -> Option D

4. Quick Check:

   Classification accuracy = Accuracy [OK]

Hint: Accuracy measures correct predictions in classification [OK]

Common Mistakes:

• Confusing regression metrics with classification
• Using BLEU for classification tasks
• Mixing Perplexity with accuracy

2. Which of the following is the correct Python syntax to calculate accuracy using scikit-learn?

easy

A. accuracy = accuracy(y_true, y_pred)

B. accuracy = score_accuracy(y_true, y_pred)

C. accuracy = accuracy_score(y_true, y_pred)

D. accuracy = calc_accuracy(y_true, y_pred)

Solution

1. Step 1: Recall scikit-learn function name

   The correct function to compute accuracy is accuracy_score from sklearn.metrics.

2. Step 2: Check function call syntax

   It requires two arguments: true labels and predicted labels, called as accuracy_score(y_true, y_pred).

3. Final Answer:

   accuracy = accuracy_score(y_true, y_pred) -> Option C

4. Quick Check:

   scikit-learn accuracy function = accuracy_score [OK]

Hint: Use accuracy_score from sklearn.metrics for accuracy [OK]

Common Mistakes:

• Using incorrect function names
• Missing import of accuracy_score
• Swapping argument order

3. Given the following code snippet, what will be the printed F1 score?

from sklearn.metrics import f1_score

y_true = [1, 0, 1, 1, 0]
y_pred = [1, 0, 0, 1, 0]
f1 = f1_score(y_true, y_pred)
print(round(f1, 2))

medium

A. 0.80

B. 0.75

C. 0.67

D. 0.60

Solution

1. Step 1: Calculate precision and recall

   True positives (TP) = 2 (positions 0 and 3), False positives (FP) = 0, False negatives (FN) = 1 (position 2).

2. Step 2: Compute F1 score

   Precision = TP / (TP + FP) = 2/2 = 1.0; Recall = TP / (TP + FN) = 2/3 ≈ 0.67; F1 = 2 * (Precision * Recall) / (Precision + Recall) ≈ 2*(1*0.67)/(1+0.67) ≈ 0.80.

3. Step 3: Verify scikit-learn default behavior

   By default, f1_score uses 'binary' average, so calculation matches above.

4. Step 4: Check rounding

   Rounded to two decimals, the printed value is 0.80, but the actual f1_score value is approximately 0.80.

5. Final Answer:

   0.80 -> Option A

6. Quick Check:

   F1 score = 0.80 [OK]

Hint: F1 balances precision and recall; calculate both first [OK]

Common Mistakes:

• Confusing precision with recall
• Rounding too early
• Ignoring default average parameter

4. You run this code but get an error:

from sklearn.metrics import precision_score

true = [1, 0, 1]
pred = [1, 1, 0]
score = precision_score(true, pred)
print(score)

What is the likely cause of the error?

medium

A. No error; code runs fine

B. Mismatch in label types causing undefined precision

C. Incorrect variable names used in function call

D. Missing import of precision_score

Solution

1. Step 1: Check imports and variables

   precision_score is imported correctly and variables true, pred are defined properly.

2. Step 2: Understand precision_score behavior

   Precision is undefined if there are no predicted positives for the positive class, which can cause warnings or errors.

3. Step 3: Analyze given data

   pred has one positive (1), true has positives at positions 0 and 2; so precision can be computed without error.

4. Step 4: Consider label types

   If labels are not binary or have unexpected types, precision_score may error; here labels are fine, so no error expected.

5. Final Answer:

   No error; code runs fine -> Option A

6. Quick Check:

   Code runs fine with correct inputs [OK]

Hint: Check label types and predicted positives for precision errors [OK]

Common Mistakes:

• Assuming import errors without checking
• Confusing variable names
• Ignoring label format requirements

5. You want to evaluate a language generation model. Which automated metric should you choose to measure how well the model's output matches human references?

hard

A. Mean Absolute Error

B. BLEU Score

C. Accuracy

D. Silhouette Score

Solution

1. Step 1: Identify task type

   Language generation models produce text outputs, so evaluation needs to compare generated text to reference text.

2. Step 2: Match metric to task

   BLEU Score measures overlap of n-grams between generated and reference text, widely used for language generation evaluation.

3. Step 3: Exclude unrelated metrics

   Mean Absolute Error is for regression, Accuracy for classification, Silhouette Score for clustering, so they don't fit language generation.

4. Final Answer:

   BLEU Score -> Option B

5. Quick Check:

   Language generation evaluation = BLEU Score [OK]

Hint: Use BLEU for comparing generated text to references [OK]

Common Mistakes:

• Using regression or classification metrics for text
• Confusing clustering metrics with language metrics
• Ignoring task-specific metric choice