Content-based filtering in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Content-based filtering

Which metric matters for Content-based filtering and WHY

Content-based filtering recommends items similar to what a user liked before. The key metric is Precision. This tells us how many recommended items are actually relevant to the user. High precision means the system suggests mostly things the user will like, avoiding annoying wrong suggestions.

Recall is also important. It measures how many of the relevant items the system finds. High recall means the system finds many good matches, not missing useful recommendations.

Balancing precision and recall is important. The F1 score combines both into one number to check overall quality.

Confusion matrix for Content-based filtering

      |-----------------------------|
      |           | Predicted       |
      | Actual    | Relevant | Not  |
      |           |         | Relevant |
      |-----------------------------|
      | Relevant  |   TP    |   FN   |
      | Not       |   FP    |   TN   |
      | Relevant  |         |        |
      |-----------------------------|

      TP = Recommended and relevant (good)
      FP = Recommended but not relevant (bad)
      FN = Not recommended but relevant (missed)
      TN = Not recommended and not relevant (neutral)
    

Precision vs Recall tradeoff with examples

If the system recommends fewer items, it may have high precision (mostly good suggestions) but low recall (missing many relevant items).

If it recommends many items, it may have high recall (finding many relevant items) but low precision (more wrong suggestions).

Example: A movie recommender that suggests only 2 movies might have high precision if both are liked, but low recall if the user likes 10 movies total. Suggesting 20 movies might catch all 10 liked ones (high recall) but also include many disliked ones (low precision).

What good vs bad metric values look like

Good: Precision and recall both above 0.7 means the system suggests mostly relevant items and finds many of them.

Bad: Precision below 0.3 means many wrong suggestions, annoying users. Recall below 0.3 means missing most relevant items, making recommendations useless.

F1 score below 0.4 usually means the system needs improvement.

Common pitfalls in metrics for Content-based filtering

• Accuracy paradox: Accuracy can be misleading if most items are irrelevant. A system that never recommends can have high accuracy but no value.
• Data leakage: Using future user preferences in training can inflate metrics unrealistically.
• Overfitting: The system may recommend only items very similar to past likes, missing diversity and hurting recall.

Self-check question

Your content-based filtering model has 98% accuracy but only 12% recall on relevant items. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most items are irrelevant, so the model is mostly not recommending anything. The very low recall means it misses almost all relevant items, so users get very few useful recommendations.

Key Result

Precision and recall are key metrics; high precision ensures relevant recommendations, high recall ensures many relevant items are found.