Evaluation metrics (RMSE, precision@k) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Evaluation metrics (RMSE, precision@k)

Which metric matters and WHY

RMSE (Root Mean Squared Error) is used when you want to measure how close your model's predicted numbers are to the actual numbers. It tells you the average size of the errors your model makes, with bigger errors counting more. This is great for tasks like predicting prices or temperatures.

Precision@k is important when you want to check how good your model is at picking the top k items that really matter. For example, if your model recommends 5 movies, precision@5 tells you how many of those 5 movies the user actually likes. This is useful in recommendation systems.

Confusion matrix or equivalent visualization

RMSE does not use a confusion matrix because it measures error size for continuous values.

Precision@k can be understood with a simple example:

    Recommended items (k=5): [A, B, C, D, E]
    Relevant items: [B, D, F, G]

    True Positives (TP) = items recommended and relevant = B, D = 2
    False Positives (FP) = items recommended but not relevant = A, C, E = 3

    Precision@5 = TP / k = 2 / 5 = 0.4
    

Precision vs Recall tradeoff with examples

Precision@k focuses on how many of the top k recommended items are actually relevant. It does not consider how many relevant items were missed.

For example, if a music app recommends 5 songs and 4 are liked (high precision), but the user actually likes 20 songs total, recall is low because many liked songs were not recommended.

RMSE focuses on how close predictions are to actual values. Lower RMSE means predictions are closer to true values.

In some cases, improving precision@k might reduce recall, and vice versa. Choosing which to focus on depends on what matters more: showing only good recommendations (precision) or showing most relevant items (recall).

What "good" vs "bad" metric values look like

RMSE: Lower values are better. For example, if predicting house prices in thousands, an RMSE of 5 means on average predictions are off by $5,000. An RMSE of 50 means much worse predictions.

Precision@k: Values range from 0 to 1. A precision@5 of 0.8 means 4 out of 5 recommended items are relevant (good). A precision@5 of 0.2 means only 1 out of 5 is relevant (bad).

Common pitfalls

• RMSE: Sensitive to outliers. A few large errors can make RMSE very high.
• Precision@k: Does not consider missed relevant items (recall). High precision but low recall means many relevant items are ignored.
• Using accuracy for recommendation or regression tasks is misleading.
• Ignoring the context: a good RMSE depends on the scale of the data.

Self-check question

Your recommendation model has a precision@5 of 0.9 but recall of 0.1. Is it good for production?

Answer: It depends on the goal. High precision@5 means most recommended items are relevant, which is good. But recall of 0.1 means it only finds 10% of all relevant items, missing many. If you want to show only very good recommendations, this might be okay. But if you want to cover more relevant items, this model misses too many.

Key Result

RMSE measures average prediction error size; precision@k measures relevance of top-k recommendations.

Practice

1. What does RMSE (Root Mean Squared Error) measure in a machine learning model?

easy

A. The average size of prediction errors for continuous values

B. The percentage of correct classifications

C. The number of true positives in classification

D. The speed of the model training process

Solution

1. Step 1: Understand RMSE definition

   RMSE calculates the square root of the average squared differences between predicted and actual values, showing error size.

2. Step 2: Identify what RMSE measures

   It measures how far predictions are from actual continuous values on average, not classification accuracy or speed.

3. Final Answer:

   The average size of prediction errors for continuous values -> Option A

4. Quick Check:

   RMSE = average error size [OK]

Hint: RMSE = error size for numbers, not classification [OK]

Common Mistakes:

• Confusing RMSE with accuracy
• Thinking RMSE measures classification counts
• Assuming RMSE measures speed

2. Which of the following is the correct formula to calculate RMSE given predictions \(\hat{y}_i\) and true values \(y_i\) for \(n\) samples?

easy

A. \(\frac{\sum_{i=1}^n (y_i - \hat{y}_i)}{n}\)

B. \(\frac{1}{n} \sum_{i=1}^n |y_i - \hat{y}_i|\)

C. \(\frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}\)

D. \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\)

Solution

1. Step 1: Recall RMSE formula

   RMSE is the square root of the average squared differences between true and predicted values.

2. Step 2: Match formula options

   \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\) matches the RMSE formula exactly; others represent MAE, mean error, or precision.

3. Final Answer:

   \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\) -> Option D

4. Quick Check:

   RMSE = sqrt(mean squared error) [OK]

Hint: RMSE = sqrt of average squared errors [OK]

Common Mistakes:

• Confusing RMSE with MAE (mean absolute error)
• Forgetting to square errors before averaging
• Mixing precision formula with RMSE

3. Given the true labels \([1, 0, 1, 1, 0]\) and model predicted scores \([0.9, 0.2, 0.8, 0.4, 0.3]\), what is the precision@3?

medium

A. 1.0

B. 0.67

C. 0.33

D. 0.5

Solution

1. Step 1: Identify top 3 predictions by score

   Top 3 scores are 0.9 (index 0), 0.8 (index 2), and 0.4 (index 3).

2. Step 2: Count true positives in top 3

   True labels at these indices are 1 (index 0), 1 (index 2), 1 (index 3). Total true positives = 3.

3. Step 3: Calculate precision@3

   Precision@3 = true positives / 3 = 3/3 = 1.0.

4. Final Answer:

   1.0 -> Option A

5. Quick Check:

   Precision@3 = 3/3 = 1.0 [OK]

Hint: Count true positives in top k predictions, divide by k [OK]

Common Mistakes:

• Counting all positives, not just top k
• Mixing precision@k with recall
• Using predicted labels instead of scores for ranking

4. You have a ranking model and calculated precision@5 as 0.4, but you expected 0.6. Which of the following could fix this issue?

medium

A. Increase the number of items in the dataset

B. Improve the model to rank more relevant items in the top 5

C. Decrease the number of true positives in the top 5

D. Use RMSE instead of precision@5

Solution

1. Step 1: Understand precision@5 meaning

   Precision@5 measures how many relevant items appear in the top 5 predictions.

2. Step 2: Identify how to increase precision@5

   To increase precision@5, the model must rank more relevant items higher within the top 5.

3. Step 3: Evaluate options

   Improve the model to rank more relevant items in the top 5 improves ranking of relevant items; A changes dataset size but not ranking; C reduces true positives, lowering precision; D changes metric but not model quality.

4. Final Answer:

   Improve the model to rank more relevant items in the top 5 -> Option B

5. Quick Check:

   Better ranking = higher precision@5 [OK]

Hint: Boost relevant items in top k to raise precision@k [OK]

Common Mistakes:

• Changing dataset size instead of model quality
• Reducing true positives mistakenly
• Confusing RMSE with precision@k

5. A regression model has predictions \([2.5, 0.0, 2.1, 7.8]\) and true values \([3.0, -0.5, 2.0, 7.5]\). Calculate the RMSE and decide if the model's average error is less than 0.5.

hard

A. RMSE = 0.65, no the error is not less than 0.5

B. RMSE = 0.55, no the error is not less than 0.5

C. RMSE = 0.39, yes the error is less than 0.5

D. RMSE = 0.35, yes the error is less than 0.5

Solution

1. Step 1: Calculate squared errors for each prediction

   (3.0-2.5)^2=0.25, (-0.5-0.0)^2=0.25, (2.0-2.1)^2=0.01, (7.5-7.8)^2=0.09. Sum=0.60.

2. Step 2: Compute mean squared error and RMSE

   Mean squared error = 0.60/4 = 0.15; RMSE = \(\sqrt{0.15}\) \(\approx\) 0.39.

3. Step 3: Compare RMSE with 0.5

   RMSE \(\approx\) 0.39 < 0.5, so average error is less than 0.5.

4. Final Answer:

   RMSE = 0.39, yes the error is less than 0.5 -> Option C

5. Quick Check:

   RMSE \(\approx\) 0.39 < 0.5 [OK]

Hint: Square errors, average, then sqrt for RMSE [OK]

Common Mistakes:

• Forgetting to square errors
• Incorrect averaging of squared errors
• Mixing RMSE with MAE