Collaborative filtering in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Collaborative filtering

Which metric matters for Collaborative filtering and WHY

Collaborative filtering predicts user preferences, so we want metrics that show how close predictions are to actual user choices.

Common metrics include:

• Root Mean Squared Error (RMSE): Measures average prediction error size. Lower is better.
• Mean Absolute Error (MAE): Average absolute difference between predicted and actual ratings.
• Precision and Recall: For top-N recommendations, precision shows how many recommended items were liked, recall shows how many liked items were recommended.
• F1 Score: Balances precision and recall for recommendation relevance.
• Mean Average Precision (MAP): Measures ranking quality of recommended items.

We choose metrics based on the goal: rating prediction accuracy or recommendation relevance.

Confusion matrix or equivalent visualization

For top-N recommendation evaluation, we can use a confusion matrix based on whether items were recommended and liked:

      |---------------------------|
      |           | Liked | Not Liked |
      |-----------|-------|----------|
      | Recommended |  TP  |    FP    |
      | Not Recommended |  FN  |    TN    |
      |---------------------------|
    

Where:

• TP (True Positive): Recommended and liked items.
• FP (False Positive): Recommended but not liked items.
• FN (False Negative): Liked but not recommended items.
• TN (True Negative): Not recommended and not liked items.

This matrix helps calculate precision, recall, and F1 for recommendations.

Precision vs Recall tradeoff with examples

In collaborative filtering, precision and recall often trade off:

• High Precision, Low Recall: Recommending fewer items but mostly liked ones. Good if you want to avoid annoying users with bad suggestions.
• High Recall, Low Precision: Recommending many items including most liked ones but also many disliked. Good if you want to show many options and don't mind some misses.

Example: A movie app wants to avoid bad recommendations (high precision). It recommends 5 movies, 4 are liked (precision=0.8), but misses many liked movies (recall low).

Another app wants to show all possible liked movies (high recall). It recommends 20 movies, 10 are liked (precision=0.5), but covers most liked movies (recall high).

What "good" vs "bad" metric values look like for Collaborative filtering

• RMSE/MAE: Good models have low values (e.g., RMSE < 1 on a 5-star scale). High values mean poor rating predictions.
• Precision: Good if above 0.7 for top recommendations. Below 0.4 means many wrong suggestions.
• Recall: Good if above 0.6, meaning most liked items are recommended. Below 0.3 means many liked items missed.
• F1 Score: Good if above 0.65, balancing precision and recall well.
• MAP: Higher values (closer to 1) mean better ranking of relevant items.

Good values depend on dataset size and user behavior but these ranges are typical starting points.

Common pitfalls in Collaborative filtering metrics

• Ignoring data sparsity: Many users rate few items, so metrics can be misleading if not averaged properly.
• Overfitting: Model performs well on training data but poorly on new users or items.
• Popularity bias: Recommending only popular items can inflate precision but reduce diversity.
• Cold start problem: New users/items have no data, metrics may be poor or undefined.
• Using accuracy alone: Accuracy is not meaningful for recommendation because most items are not interacted with.
• Data leakage: Testing on data that the model has seen during training inflates metrics falsely.

Self-check question

Your collaborative filtering model has 98% accuracy but only 12% recall on liked items. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most items are not liked, so predicting "not liked" often is easy. The low recall means the model misses most liked items, so users get poor recommendations.

Key Result

Precision, recall, and RMSE are key metrics to evaluate collaborative filtering models, balancing recommendation relevance and prediction accuracy.

Practice

1. What is the main idea behind collaborative filtering in recommendation systems?

easy

A. It recommends items based on the user's location.

B. It recommends items based on similar users' preferences.

C. It randomly suggests items without user data.

D. It uses item descriptions to recommend similar items.

Solution

1. Step 1: Understand collaborative filtering concept

   Collaborative filtering finds users with similar tastes and recommends items they liked.

2. Step 2: Compare options to definition

   Only It recommends items based on similar users' preferences. matches this idea; others describe different methods or unrelated ideas.

3. Final Answer:

   It recommends items based on similar users' preferences. -> Option B

4. Quick Check:

   Collaborative filtering = similar users' preferences [OK]

Hint: Focus on user similarity for collaborative filtering [OK]

Common Mistakes:

• Confusing collaborative filtering with content-based filtering
• Thinking it uses item descriptions only
• Assuming it recommends randomly

2. Which of the following is the correct way to compute cosine similarity between two user vectors u and v in Python using NumPy?

easy

A. np.dot(u, v) * (np.linalg.norm(u) + np.linalg.norm(v))

B. np.dot(u, v) * (np.linalg.norm(u) * np.linalg.norm(v))

C. np.dot(u, v) / (np.linalg.norm(u) + np.linalg.norm(v))

D. np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))

Solution

1. Step 1: Recall cosine similarity formula

   Cosine similarity = dot product of vectors divided by product of their norms.

2. Step 2: Match formula to code

   np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) correctly implements this formula using NumPy functions.

3. Final Answer:

   np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) -> Option D

4. Quick Check:

   Cosine similarity formula = np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) [OK]

Hint: Cosine similarity = dot product ÷ product of norms [OK]

Common Mistakes:

• Adding norms instead of multiplying
• Multiplying dot product by norms
• Dividing by sum of norms

3. Given the user-item rating matrix below, what is the predicted rating for User 1 on Item C using user-based collaborative filtering with cosine similarity?

User 1: [5, 3, ?]
User 2: [4, 2, 1]
User 3: [1, 5, 4]

Assume similarity weights are:
User 1 & User 2: 0.9
User 1 & User 3: 0.1

Use weighted average: (0.9*1 + 0.1*4) / (0.9 + 0.1)

medium

A. 2.5

B. 1.9

C. 1.3

D. 3.0

Solution

1. Step 1: Calculate weighted sum of ratings

   Weighted sum = (0.9 * 1) + (0.1 * 4) = 0.9 + 0.4 = 1.3

2. Step 2: Calculate sum of weights and divide

   Sum weights = 0.9 + 0.1 = 1.0; Predicted rating = 1.3 / 1.0 = 1.3

3. Final Answer:

   1.3 -> Option C

4. Quick Check:

   Weighted average = 1.3 [OK]

Hint: Multiply ratings by similarity, sum, then divide by total similarity [OK]

Common Mistakes:

• Forgetting to divide by sum of weights
• Mixing up user similarities
• Using unweighted average

4. You have this Python code snippet for item-based collaborative filtering similarity calculation:

import numpy as np

item1 = np.array([5, 3, 0])
item2 = np.array([4, 0, 1])
similarity = np.dot(item1, item2) / (np.linalg.norm(item1) + np.linalg.norm(item2))
print(similarity)

What is the error in this code?

medium

A. Using addition instead of multiplication in denominator for cosine similarity.

B. Using np.dot instead of np.cross for similarity.

C. Missing normalization of item vectors before dot product.

D. Using zero values in vectors causes division by zero.

Solution

1. Step 1: Review cosine similarity formula

   Cosine similarity divides dot product by product of norms, not sum.

2. Step 2: Identify code mistake

   Code uses addition in denominator instead of multiplication, causing wrong similarity value.

3. Final Answer:

   Using addition instead of multiplication in denominator for cosine similarity. -> Option A

4. Quick Check:

   Denominator must multiply norms, not add [OK]

Hint: Denominator in cosine similarity = product of norms, not sum [OK]

Common Mistakes:

• Adding norms instead of multiplying
• Confusing dot and cross product
• Assuming zero values cause errors

5. You want to improve a movie recommendation system using collaborative filtering. The user-item matrix is very sparse (many missing ratings). Which approach can best handle this sparsity to improve recommendations?

hard

A. Use matrix factorization techniques like Singular Value Decomposition (SVD).

B. Fill missing ratings with zeros before computing similarities.

C. Ignore users with missing ratings and use only complete data.

D. Use random recommendations to cover missing data.

Solution

1. Step 1: Understand sparsity problem in collaborative filtering

   Sparse matrices have many missing values, making similarity calculations unreliable.

2. Step 2: Identify best method to handle sparsity

   Matrix factorization like SVD reduces dimensionality and predicts missing ratings effectively.

3. Final Answer:

   Use matrix factorization techniques like Singular Value Decomposition (SVD). -> Option A

4. Quick Check:

   SVD handles sparsity well [OK]

Hint: Use matrix factorization to predict missing ratings [OK]

Common Mistakes:

• Filling missing ratings with zeros biases similarity
• Ignoring incomplete users loses valuable data
• Random recommendations ignore user preferences