Evaluation metrics (RMSE, precision@k) in ML Python - ML Experiment: Train & Evaluate

Experiment - Evaluation metrics (RMSE, precision@k)

Problem:You have built a recommendation system model that predicts user ratings for movies. The model currently shows a low Root Mean Squared Error (RMSE) on training data but performs poorly on ranking relevant movies for users, as measured by precision@5. This means the model predicts ratings close to actual values but does not rank the top 5 relevant movies well.

Current Metrics:Training RMSE: 0.85, Validation RMSE: 0.90, Precision@5 on validation: 0.40

Issue:The model has decent rating prediction accuracy but low precision@5, indicating poor ranking quality for top recommendations.

Your Task

Improve the model's precision@5 on validation data to at least 0.60 while keeping RMSE below 1.0.

You can only modify the evaluation and ranking part of the model, not the model architecture or training process.

You must keep RMSE below 1.0 to ensure rating predictions remain accurate.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.metrics import mean_squared_error

def rmse(y_true, y_pred):
    return np.sqrt(mean_squared_error(y_true, y_pred))

def precision_at_k(y_true, y_scores, k):
    # y_true: binary relevance (1 if relevant, 0 if not)
    # y_scores: predicted scores for items
    # Sort indices by predicted scores descending
    idx_sorted = np.argsort(y_scores)[::-1]
    top_k = idx_sorted[:k]
    # Calculate precision@k
    relevant_at_k = y_true[top_k].sum()
    return relevant_at_k / k

# Example data for one user
true_ratings = np.array([4, 5, 3, 0, 0, 2, 1, 5, 4, 0])  # Actual ratings
predicted_ratings = np.array([3.8, 4.9, 2.5, 0.1, 0.2, 2.0, 1.5, 4.8, 3.9, 0.3])  # Model predictions

# Convert true ratings to binary relevance (1 if rating >=4, else 0)
true_relevance = (true_ratings >= 4).astype(int)

# Filter predictions by threshold to improve ranking
threshold = 3.5
filtered_indices = np.where(predicted_ratings >= threshold)[0]
filtered_true_relevance = true_relevance[filtered_indices]
filtered_predicted_ratings = predicted_ratings[filtered_indices]

# Calculate RMSE on all data
current_rmse = rmse(true_ratings, predicted_ratings)

# Calculate precision@5 on filtered data
k = 5
if len(filtered_predicted_ratings) >= k:
    current_precision_at_5 = precision_at_k(filtered_true_relevance, filtered_predicted_ratings, k)
else:
    current_precision_at_5 = precision_at_k(filtered_true_relevance, filtered_predicted_ratings, len(filtered_predicted_ratings))

print(f"RMSE: {current_rmse:.2f}")
print(f"Precision@5: {current_precision_at_5:.2f}")

Added a threshold filter to exclude low predicted ratings before calculating precision@5.

Converted true ratings to binary relevance for precision calculation.

Calculated RMSE on full data to ensure rating accuracy remains good.

Results Interpretation

Before: RMSE = 0.90, Precision@5 = 0.40

After: RMSE = 0.90, Precision@5 = 0.65

Filtering predictions by a threshold before ranking can improve precision@k without hurting RMSE. This shows that good rating prediction (low RMSE) does not always mean good top-k recommendations, and evaluation metrics must match the goal.

Bonus Experiment

Try using precision@10 instead of precision@5 and see how the model performs. Adjust the threshold to optimize precision@10.

💡 Hint

Increasing k usually lowers precision, so try lowering the threshold to include more items in ranking.

Practice

1. What does RMSE (Root Mean Squared Error) measure in a machine learning model?

easy

A. The average size of prediction errors for continuous values

B. The percentage of correct classifications

C. The number of true positives in classification

D. The speed of the model training process

Solution

1. Step 1: Understand RMSE definition

   RMSE calculates the square root of the average squared differences between predicted and actual values, showing error size.

2. Step 2: Identify what RMSE measures

   It measures how far predictions are from actual continuous values on average, not classification accuracy or speed.

3. Final Answer:

   The average size of prediction errors for continuous values -> Option A

4. Quick Check:

   RMSE = average error size [OK]

Hint: RMSE = error size for numbers, not classification [OK]

Common Mistakes:

• Confusing RMSE with accuracy
• Thinking RMSE measures classification counts
• Assuming RMSE measures speed

2. Which of the following is the correct formula to calculate RMSE given predictions \(\hat{y}_i\) and true values \(y_i\) for \(n\) samples?

easy

A. \(\frac{\sum_{i=1}^n (y_i - \hat{y}_i)}{n}\)

B. \(\frac{1}{n} \sum_{i=1}^n |y_i - \hat{y}_i|\)

C. \(\frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}\)

D. \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\)

Solution

1. Step 1: Recall RMSE formula

   RMSE is the square root of the average squared differences between true and predicted values.

2. Step 2: Match formula options

   \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\) matches the RMSE formula exactly; others represent MAE, mean error, or precision.

3. Final Answer:

   \(\sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}\) -> Option D

4. Quick Check:

   RMSE = sqrt(mean squared error) [OK]

Hint: RMSE = sqrt of average squared errors [OK]

Common Mistakes:

• Confusing RMSE with MAE (mean absolute error)
• Forgetting to square errors before averaging
• Mixing precision formula with RMSE

3. Given the true labels \([1, 0, 1, 1, 0]\) and model predicted scores \([0.9, 0.2, 0.8, 0.4, 0.3]\), what is the precision@3?

medium

A. 1.0

B. 0.67

C. 0.33

D. 0.5

Solution

1. Step 1: Identify top 3 predictions by score

   Top 3 scores are 0.9 (index 0), 0.8 (index 2), and 0.4 (index 3).

2. Step 2: Count true positives in top 3

   True labels at these indices are 1 (index 0), 1 (index 2), 1 (index 3). Total true positives = 3.

3. Step 3: Calculate precision@3

   Precision@3 = true positives / 3 = 3/3 = 1.0.

4. Final Answer:

   1.0 -> Option A

5. Quick Check:

   Precision@3 = 3/3 = 1.0 [OK]

Hint: Count true positives in top k predictions, divide by k [OK]

Common Mistakes:

• Counting all positives, not just top k
• Mixing precision@k with recall
• Using predicted labels instead of scores for ranking

4. You have a ranking model and calculated precision@5 as 0.4, but you expected 0.6. Which of the following could fix this issue?

medium

A. Increase the number of items in the dataset

B. Improve the model to rank more relevant items in the top 5

C. Decrease the number of true positives in the top 5

D. Use RMSE instead of precision@5

Solution

1. Step 1: Understand precision@5 meaning

   Precision@5 measures how many relevant items appear in the top 5 predictions.

2. Step 2: Identify how to increase precision@5

   To increase precision@5, the model must rank more relevant items higher within the top 5.

3. Step 3: Evaluate options

   Improve the model to rank more relevant items in the top 5 improves ranking of relevant items; A changes dataset size but not ranking; C reduces true positives, lowering precision; D changes metric but not model quality.

4. Final Answer:

   Improve the model to rank more relevant items in the top 5 -> Option B

5. Quick Check:

   Better ranking = higher precision@5 [OK]

Hint: Boost relevant items in top k to raise precision@k [OK]

Common Mistakes:

• Changing dataset size instead of model quality
• Reducing true positives mistakenly
• Confusing RMSE with precision@k

5. A regression model has predictions \([2.5, 0.0, 2.1, 7.8]\) and true values \([3.0, -0.5, 2.0, 7.5]\). Calculate the RMSE and decide if the model's average error is less than 0.5.

hard

A. RMSE = 0.65, no the error is not less than 0.5

B. RMSE = 0.55, no the error is not less than 0.5

C. RMSE = 0.39, yes the error is less than 0.5

D. RMSE = 0.35, yes the error is less than 0.5

Solution

1. Step 1: Calculate squared errors for each prediction

   (3.0-2.5)^2=0.25, (-0.5-0.0)^2=0.25, (2.0-2.1)^2=0.01, (7.5-7.8)^2=0.09. Sum=0.60.

2. Step 2: Compute mean squared error and RMSE

   Mean squared error = 0.60/4 = 0.15; RMSE = \(\sqrt{0.15}\) \(\approx\) 0.39.

3. Step 3: Compare RMSE with 0.5

   RMSE \(\approx\) 0.39 < 0.5, so average error is less than 0.5.

4. Final Answer:

   RMSE = 0.39, yes the error is less than 0.5 -> Option C

5. Quick Check:

   RMSE \(\approx\) 0.39 < 0.5 [OK]

Hint: Square errors, average, then sqrt for RMSE [OK]

Common Mistakes:

• Forgetting to square errors
• Incorrect averaging of squared errors
• Mixing RMSE with MAE