Why recommendations drive engagement in ML Python - Why Metrics Matter

Metrics & Evaluation - Why recommendations drive engagement

Which metric matters for this concept and WHY

For recommendation systems, the key metrics are Precision, Recall, and F1 score. These metrics tell us how well the system suggests items users actually like or engage with. High precision means most recommended items are relevant, so users see good suggestions. High recall means the system finds most of the items users would like, so it doesn't miss good options. F1 balances both, showing overall quality. Engagement depends on showing relevant items without overwhelming users with bad suggestions.

Confusion matrix or equivalent visualization (ASCII)

    Recommended Items vs User Likes

               | Liked (Positive) | Not Liked (Negative)
    -----------|------------------|---------------------
    Recommended|       TP         |         FP          
    Not Rec.   |       FN         |         TN          

    TP = Recommended and liked (good)
    FP = Recommended but not liked (bad)
    FN = Not recommended but liked (missed)
    TN = Not recommended and not liked (neutral)
    

Precision = TP / (TP + FP) measures how many recommended items were actually liked.

Recall = TP / (TP + FN) measures how many liked items were recommended.

Precision vs Recall tradeoff with concrete examples

Imagine a movie app recommending films:

• High Precision, Low Recall: The app shows only a few movies it is very sure the user will like. Users see mostly good picks but might miss many other movies they would enjoy.
• High Recall, Low Precision: The app shows many movies, including most the user would like, but also many irrelevant ones. Users get overwhelmed and may lose trust.

Good recommendations balance precision and recall to keep users engaged by showing relevant items without too much noise.

What "good" vs "bad" metric values look like for this use case

Good metrics:

• Precision around 0.7 or higher means most recommended items are relevant.
• Recall around 0.6 or higher means the system finds many items the user likes.
• F1 score above 0.65 shows balanced performance.

Bad metrics:

• Precision below 0.4 means many irrelevant recommendations, frustrating users.
• Recall below 0.3 means the system misses most items users would like.
• F1 score below 0.4 indicates poor overall recommendation quality.

Metrics pitfalls

• Accuracy paradox: Accuracy can be misleading if most items are not liked. A system recommending nothing can have high accuracy but zero usefulness.
• Data leakage: Using future user behavior in training can inflate metrics but fail in real use.
• Overfitting: Model performs well on training data but poorly on new users, showing high metrics that don't reflect real engagement.

Self-check question

Your recommendation model has 98% accuracy but only 12% recall on liked items. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most items are not liked, so the model just avoids recommending. The very low recall means it misses almost all items users would like, so it won't drive engagement.

Key Result

Precision and recall are key to measuring recommendation quality; balanced values drive user engagement.

Practice

1. Why do recommendation systems increase user engagement on websites or apps?

easy

A. They show users content or products that match their interests

B. They randomly display items without any user data

C. They reduce the number of items users can see

D. They force users to see ads only

Solution

1. Step 1: Understand user engagement

   User engagement means users spend more time and interact more with the platform.

2. Step 2: Role of recommendations

   Recommendations show users items they like, making them more interested and active.

3. Final Answer:

   They show users content or products that match their interests -> Option A

4. Quick Check:

   Relevant content = higher engagement [OK]

Hint: Recommendations match user interests to boost engagement [OK]

Common Mistakes:

• Thinking recommendations show random items
• Believing recommendations limit user choices
• Confusing ads with recommendations

2. Which of the following is the correct way to describe a recommendation system in simple terms?

easy

A. A system that shows the same items to all users

B. A system that guesses what users might like based on their past behavior

C. A system that deletes user data after each session

D. A system that only shows the newest items regardless of user interest

Solution

1. Step 1: Define recommendation system

   It uses past user data to predict what they might like next.

2. Step 2: Check options

   Only the system that guesses what users might like based on their past behavior describes this behavior correctly; others ignore personalization.

3. Final Answer:

   A system that guesses what users might like based on their past behavior -> Option B

4. Quick Check:

   Personalization = guessing user likes [OK]

Hint: Recommendations use past data to guess user preferences [OK]

Common Mistakes:

• Thinking recommendations show same items to everyone
• Confusing recommendation with data deletion
• Ignoring user history in recommendations

3. Consider this Python code snippet for a simple recommendation score calculation:

user_preferences = {'sports': 5, 'music': 3, 'movies': 4}
item_features = {'sports': 1, 'music': 0, 'movies': 1}
score = sum(user_preferences[cat] * item_features[cat] for cat in item_features)
print(score)

What will be the output of this code?

medium

A. 8

B. 10

C. 7

D. 9

Solution

1. Step 1: Calculate weighted scores

   Multiply user preference by item feature for each category: sports=5*1=5, music=3*0=0, movies=4*1=4.

2. Step 2: Sum the results

   Sum = 5 + 0 + 4 = 9.

3. Final Answer:

   9 -> Option D

4. Quick Check:

   5+0+4 = 9 [OK]

Hint: Multiply and sum matching categories for score [OK]

Common Mistakes:

• Adding all preference values without weights
• Ignoring zero feature values
• Multiplying wrong pairs of categories

4. The following code is intended to calculate a recommendation score but has a bug:

user_prefs = {'books': 4, 'games': 2}
item_feats = {'books': 1, 'games': 1}
score = 0
for cat in user_prefs:
    score += user_prefs[cat] * item_feats[cat]
print(score)

What is the error and how to fix it?

medium

A. KeyError if item_feats lacks a category; fix by iterating item_feats keys

B. SyntaxError due to missing colon; fix by adding colon after for loop

C. TypeError because score is int; fix by converting score to float

D. No error; code runs correctly

Solution

1. Step 1: Identify potential KeyError

   The loop uses keys from user_prefs but accesses item_feats; if item_feats misses a key, error occurs.

2. Step 2: Fix iteration to avoid missing keys

   Iterate over keys in item_feats or check key existence before access.

3. Final Answer:

   KeyError if item_feats lacks a category; fix by iterating item_feats keys -> Option A

4. Quick Check:

   Key mismatch causes error [OK]

Hint: Check keys exist before accessing dict values [OK]

Common Mistakes:

• Assuming all keys match in both dicts
• Ignoring possible KeyError exceptions
• Confusing syntax errors with runtime errors

5. A streaming service wants to improve engagement by recommending movies. They have user watch history and movie genres. Which approach best uses machine learning to drive engagement?

hard

A. Display only movies with the highest global ratings ignoring user data

B. Recommend only the newest movies regardless of user preferences

C. Build a model that predicts user ratings for unseen movies based on past watched genres

D. Show random movies from the entire catalog to all users

Solution

1. Step 1: Understand personalized recommendations

   Using user watch history and genres helps predict what each user likes.

2. Step 2: Evaluate options

   Build a model that predicts user ratings for unseen movies based on past watched genres uses machine learning to personalize, increasing engagement; others ignore personalization.

3. Final Answer:

   Build a model that predicts user ratings for unseen movies based on past watched genres -> Option C

4. Quick Check:

   Personalized prediction = better engagement [OK]

Hint: Use user history to predict preferences for recommendations [OK]

Common Mistakes:

• Ignoring user preferences in recommendations
• Assuming newest or highest rated always engage users
• Confusing random recommendations with personalized ones