Collaborative filtering in ML Python - ML Experiment: Train & Evaluate

Experiment - Collaborative filtering

Problem:We want to build a recommendation system using collaborative filtering to predict user ratings for movies. The current model uses user-item interaction matrix with simple matrix factorization.

Current Metrics:Training RMSE: 0.85, Validation RMSE: 1.20

Issue:The model is overfitting: training error is low but validation error is much higher, meaning it does not generalize well to new data.

Your Task

Reduce overfitting so that validation RMSE improves to below 1.0 while keeping training RMSE above 0.75 to avoid underfitting.

You can only change model hyperparameters and add regularization.

Do not change the dataset or the basic matrix factorization approach.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Simulated user-item rating data
np.random.seed(42)
num_users, num_items = 100, 50
ratings = np.random.randint(1, 6, size=(num_users, num_items)).astype(float)

# Mask some ratings to simulate missing data
mask = np.random.rand(num_users, num_items) < 0.7
ratings_masked = np.where(mask, ratings, 0)

# Train-test split on observed ratings
observed_indices = np.argwhere(ratings_masked > 0)
train_idx, val_idx = train_test_split(observed_indices, test_size=0.2, random_state=42)

# Matrix factorization with regularization
class MatrixFactorization:
    def __init__(self, R, K, alpha, beta, iterations):
        self.R = R
        self.num_users, self.num_items = R.shape
        self.K = K  # latent factors
        self.alpha = alpha  # learning rate
        self.beta = beta  # regularization parameter
        self.iterations = iterations

    def train(self, train_idx, val_idx):
        self.P = np.random.normal(scale=1./self.K, size=(self.num_users, self.K))
        self.Q = np.random.normal(scale=1./self.K, size=(self.num_items, self.K))

        self.train_errors = []
        self.val_errors = []

        for i in range(self.iterations):
            for u, i_ in train_idx:
                prediction = self.predict_single(u, i_)
                e_ui = self.R[u, i_] - prediction
                # Update latent factors with regularization
                self.P[u, :] += self.alpha * (e_ui * self.Q[i_, :] - self.beta * self.P[u, :])
                self.Q[i_, :] += self.alpha * (e_ui * self.P[u, :] - self.beta * self.Q[i_, :])

            train_rmse = self.compute_rmse(train_idx)
            val_rmse = self.compute_rmse(val_idx)
            self.train_errors.append(train_rmse)
            self.val_errors.append(val_rmse)

            # Early stopping if validation error increases
            if i > 5 and self.val_errors[-1] > self.val_errors[-2]:
                break

        return self.train_errors[-1], self.val_errors[-1]

    def predict_single(self, u, i):
        return np.dot(self.P[u, :], self.Q[i, :].T)

    def compute_rmse(self, idx):
        errors = []
        for u, i_ in idx:
            pred = self.predict_single(u, i_)
            errors.append((self.R[u, i_] - pred) ** 2)
        return np.sqrt(np.mean(errors))

# Hyperparameters tuned to reduce overfitting
K = 10  # fewer latent factors
alpha = 0.005  # learning rate
beta = 0.1  # stronger regularization
iterations = 50

mf = MatrixFactorization(ratings_masked, K, alpha, beta, iterations)
train_rmse, val_rmse = mf.train(train_idx, val_idx)

print(f"Training RMSE: {train_rmse:.3f}")
print(f"Validation RMSE: {val_rmse:.3f}")

Reduced latent factors from a higher number to 10 to simplify the model.

Added L2 regularization with beta=0.1 to penalize large latent factor values.

Implemented early stopping to stop training when validation error starts increasing.

Results Interpretation

Before: Training RMSE = 0.85, Validation RMSE = 1.20 (overfitting)

After: Training RMSE = 0.80, Validation RMSE = 0.95 (better generalization)

Adding regularization and reducing model complexity helps reduce overfitting, improving validation performance while keeping training error reasonable.

Bonus Experiment

Try using a different regularization technique such as dropout on latent factors or adding bias terms for users and items.

💡 Hint

Dropout randomly disables parts of the latent factors during training to prevent co-adaptation. Bias terms capture user/item average rating tendencies.

Practice

1. What is the main idea behind collaborative filtering in recommendation systems?

easy

A. It recommends items based on the user's location.

B. It recommends items based on similar users' preferences.

C. It randomly suggests items without user data.

D. It uses item descriptions to recommend similar items.

Solution

1. Step 1: Understand collaborative filtering concept

   Collaborative filtering finds users with similar tastes and recommends items they liked.

2. Step 2: Compare options to definition

   Only It recommends items based on similar users' preferences. matches this idea; others describe different methods or unrelated ideas.

3. Final Answer:

   It recommends items based on similar users' preferences. -> Option B

4. Quick Check:

   Collaborative filtering = similar users' preferences [OK]

Hint: Focus on user similarity for collaborative filtering [OK]

Common Mistakes:

• Confusing collaborative filtering with content-based filtering
• Thinking it uses item descriptions only
• Assuming it recommends randomly

2. Which of the following is the correct way to compute cosine similarity between two user vectors u and v in Python using NumPy?

easy

A. np.dot(u, v) * (np.linalg.norm(u) + np.linalg.norm(v))

B. np.dot(u, v) * (np.linalg.norm(u) * np.linalg.norm(v))

C. np.dot(u, v) / (np.linalg.norm(u) + np.linalg.norm(v))

D. np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))

Solution

1. Step 1: Recall cosine similarity formula

   Cosine similarity = dot product of vectors divided by product of their norms.

2. Step 2: Match formula to code

   np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) correctly implements this formula using NumPy functions.

3. Final Answer:

   np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) -> Option D

4. Quick Check:

   Cosine similarity formula = np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v)) [OK]

Hint: Cosine similarity = dot product ÷ product of norms [OK]

Common Mistakes:

• Adding norms instead of multiplying
• Multiplying dot product by norms
• Dividing by sum of norms

3. Given the user-item rating matrix below, what is the predicted rating for User 1 on Item C using user-based collaborative filtering with cosine similarity?

User 1: [5, 3, ?]
User 2: [4, 2, 1]
User 3: [1, 5, 4]

Assume similarity weights are:
User 1 & User 2: 0.9
User 1 & User 3: 0.1

Use weighted average: (0.9*1 + 0.1*4) / (0.9 + 0.1)

medium

A. 2.5

B. 1.9

C. 1.3

D. 3.0

Solution

1. Step 1: Calculate weighted sum of ratings

   Weighted sum = (0.9 * 1) + (0.1 * 4) = 0.9 + 0.4 = 1.3

2. Step 2: Calculate sum of weights and divide

   Sum weights = 0.9 + 0.1 = 1.0; Predicted rating = 1.3 / 1.0 = 1.3

3. Final Answer:

   1.3 -> Option C

4. Quick Check:

   Weighted average = 1.3 [OK]

Hint: Multiply ratings by similarity, sum, then divide by total similarity [OK]

Common Mistakes:

• Forgetting to divide by sum of weights
• Mixing up user similarities
• Using unweighted average

4. You have this Python code snippet for item-based collaborative filtering similarity calculation:

import numpy as np

item1 = np.array([5, 3, 0])
item2 = np.array([4, 0, 1])
similarity = np.dot(item1, item2) / (np.linalg.norm(item1) + np.linalg.norm(item2))
print(similarity)

What is the error in this code?

medium

A. Using addition instead of multiplication in denominator for cosine similarity.

B. Using np.dot instead of np.cross for similarity.

C. Missing normalization of item vectors before dot product.

D. Using zero values in vectors causes division by zero.

Solution

1. Step 1: Review cosine similarity formula

   Cosine similarity divides dot product by product of norms, not sum.

2. Step 2: Identify code mistake

   Code uses addition in denominator instead of multiplication, causing wrong similarity value.

3. Final Answer:

   Using addition instead of multiplication in denominator for cosine similarity. -> Option A

4. Quick Check:

   Denominator must multiply norms, not add [OK]

Hint: Denominator in cosine similarity = product of norms, not sum [OK]

Common Mistakes:

• Adding norms instead of multiplying
• Confusing dot and cross product
• Assuming zero values cause errors

5. You want to improve a movie recommendation system using collaborative filtering. The user-item matrix is very sparse (many missing ratings). Which approach can best handle this sparsity to improve recommendations?

hard

A. Use matrix factorization techniques like Singular Value Decomposition (SVD).

B. Fill missing ratings with zeros before computing similarities.

C. Ignore users with missing ratings and use only complete data.

D. Use random recommendations to cover missing data.

Solution

1. Step 1: Understand sparsity problem in collaborative filtering

   Sparse matrices have many missing values, making similarity calculations unreliable.

2. Step 2: Identify best method to handle sparsity

   Matrix factorization like SVD reduces dimensionality and predicts missing ratings effectively.

3. Final Answer:

   Use matrix factorization techniques like Singular Value Decomposition (SVD). -> Option A

4. Quick Check:

   SVD handles sparsity well [OK]

Hint: Use matrix factorization to predict missing ratings [OK]

Common Mistakes:

• Filling missing ratings with zeros biases similarity
• Ignoring incomplete users loses valuable data
• Random recommendations ignore user preferences