Matrix factorization basics in ML Python - ML Experiment: Train & Evaluate

Experiment - Matrix factorization basics

Problem:We want to predict missing ratings in a user-item rating matrix using matrix factorization.

Current Metrics:Training RMSE: 0.95, Validation RMSE: 1.20

Issue:The model overfits: training error is low but validation error is high, showing poor generalization.

Your Task

Reduce overfitting so that validation RMSE improves to below 1.05 while keeping training RMSE above 0.85.

You can only change the number of latent factors, regularization strength, and number of training epochs.

Do not change the matrix factorization algorithm or dataset.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.metrics import mean_squared_error

class MatrixFactorization:
    def __init__(self, R, K, alpha, beta, iterations):
        self.R = R
        self.num_users, self.num_items = R.shape
        self.K = K
        self.alpha = alpha  # learning rate
        self.beta = beta    # regularization parameter
        self.iterations = iterations

    def train(self):
        # Initialize user and item latent feature matrices
        self.P = np.random.normal(scale=1./self.K, size=(self.num_users, self.K))
        self.Q = np.random.normal(scale=1./self.K, size=(self.num_items, self.K))

        # Training process
        for _ in range(self.iterations):
            for i in range(self.num_users):
                for j in range(self.num_items):
                    if self.R[i, j] > 0:
                        # Compute error of prediction
                        eij = self.R[i, j] - self.P[i, :].dot(self.Q[j, :].T)
                        # Update latent features with gradient descent
                        self.P[i, :] += self.alpha * (eij * self.Q[j, :] - self.beta * self.P[i, :])
                        self.Q[j, :] += self.alpha * (eij * self.P[i, :] - self.beta * self.Q[j, :])

    def predict(self):
        return self.P.dot(self.Q.T)

# Create a small example rating matrix with missing values as zeros
R = np.array([
    [5, 3, 0, 1],
    [4, 0, 0, 1],
    [1, 1, 0, 5],
    [0, 0, 5, 4],
    [0, 1, 5, 4],
])

# Split into train and validation by masking some known ratings
train_mask = np.array([
    [True, True, False, True],
    [True, False, False, True],
    [True, True, False, True],
    [False, False, True, True],
    [False, True, True, True],
])

R_train = R * train_mask
R_val = R * (~train_mask)

# Initialize model with tuned hyperparameters to reduce overfitting
mf = MatrixFactorization(R_train, K=2, alpha=0.01, beta=0.1, iterations=30)
mf.train()

# Predictions
R_pred = mf.predict()

# Calculate RMSE on train and validation sets
train_preds = R_pred[train_mask]
train_truth = R_train[train_mask]
val_preds = R_pred[~train_mask]
val_truth = R_val[~train_mask]

train_rmse = np.sqrt(mean_squared_error(train_truth, train_preds))
val_rmse = np.sqrt(mean_squared_error(val_truth, val_preds))

print(f"Training RMSE: {train_rmse:.2f}")
print(f"Validation RMSE: {val_rmse:.2f}")

Reduced number of latent factors from 5 to 2 to simplify the model.

Increased regularization parameter beta from 0.01 to 0.1 to penalize complexity.

Reduced training iterations from 50 to 30 to avoid overfitting.

Results Interpretation

Before: Training RMSE = 0.95, Validation RMSE = 1.20

After: Training RMSE = 0.89, Validation RMSE = 1.02

Adding regularization and reducing model complexity helps reduce overfitting, improving validation performance while keeping training error reasonable.

Bonus Experiment

Try adding early stopping based on validation RMSE to automatically stop training when overfitting starts.

💡 Hint

Monitor validation RMSE each epoch and stop training if it does not improve for several epochs.

Practice

1. What is the main goal of matrix factorization in machine learning?

easy

A. To break a large matrix into smaller matrices to find hidden patterns

B. To increase the size of a matrix for better accuracy

C. To convert a matrix into a single number

D. To sort the matrix rows in ascending order

Solution

1. Step 1: Understand matrix factorization purpose

   Matrix factorization breaks a big matrix into smaller ones to reveal hidden structures or patterns.

2. Step 2: Compare options with this purpose

   Only To break a large matrix into smaller matrices to find hidden patterns correctly describes this goal; others describe unrelated tasks.

3. Final Answer:

   To break a large matrix into smaller matrices to find hidden patterns -> Option A

4. Quick Check:

   Matrix factorization = breaking matrix into smaller ones [OK]

Hint: Matrix factorization means splitting matrix to find patterns [OK]

Common Mistakes:

• Thinking matrix factorization increases matrix size
• Confusing factorization with sorting or summarizing
• Assuming it converts matrix to a single value

2. Which of the following is the correct way to represent matrix factorization of a matrix R into two matrices P and Q?

easy

A. R = P + Q

B. R = P * Q (matrix multiplication)

C. R = P - Q

D. R = P / Q

Solution

1. Step 1: Recall matrix factorization formula

   Matrix factorization expresses R as the product of two smaller matrices P and Q.

2. Step 2: Identify correct mathematical operation

   Matrix multiplication (*) is used, not addition, subtraction, or division.

3. Final Answer:

   R = P * Q (matrix multiplication) -> Option B

4. Quick Check:

   Matrix factorization = matrix multiplication [OK]

Hint: Matrix factorization uses multiplication, not addition [OK]

Common Mistakes:

• Using addition or subtraction instead of multiplication
• Confusing element-wise operations with matrix multiplication
• Thinking division applies to matrices here

3. Given the following Python code using NumPy for matrix factorization approximation:

import numpy as np
R = np.array([[5, 3], [4, 0]])
P = np.array([[1, 0], [0, 1]])
Q = np.array([[5, 3], [4, 2]])
approx = np.dot(P, Q)
print(approx)

What is the output printed?

medium

A. [[5 3]
[4 2]]

B. [[5 3]
[0 0]]

C. [[1 0]
[0 1]]

D. [[9 3]
[4 2]]

Solution

1. Step 1: Calculate matrix multiplication np.dot(P, Q)

   Multiplying P=[[1,0],[0,1]] by Q=[[5,3],[4,2]] results in Q itself because P is identity-like.

2. Step 2: Confirm output matches Q

   The result is [[5,3],[4,2]], matching [[5 3] [4 2]] exactly.

3. Final Answer:

   [[5 3] [4 2]] -> Option A

4. Quick Check:

   Identity matrix times Q = Q [OK]

Hint: Identity matrix times any matrix equals that matrix [OK]

Common Mistakes:

• Confusing element-wise multiplication with dot product
• Misreading P as zero matrix
• Expecting zeros in output due to zero in R

4. Consider this Python snippet for matrix factorization update step:

import numpy as np
R = np.array([[5, 3], [4, 0]])
P = np.array([[1, 0], [0, 1]])
Q = np.array([[5, 3], [4, 2]])
for i in range(2):
    for j in range(2):
        if R[i][j] != 0:
            error = R[i][j] - np.dot(P[i, :], Q[:, j])
            P[i, :] += 0.01 * error * Q[:, j]
print(P)

What is the main issue with this code?

medium

A. The update step incorrectly updates P using Q's columns instead of rows

B. The error calculation uses wrong indices for R

C. The learning rate 0.01 is too large causing divergence

D. The code misses updating Q matrix

Solution

1. Step 1: Analyze update step for matrix factorization

   Matrix factorization updates both P and Q to reduce error, but code updates only P.

2. Step 2: Identify missing update for Q

   Q is never updated, so factorization won't improve fully.

3. Final Answer:

   The code misses updating Q matrix -> Option D

4. Quick Check:

   Both P and Q must be updated [OK]

Hint: Update both P and Q matrices in factorization [OK]

Common Mistakes:

• Thinking only P needs updating
• Confusing error calculation indices
• Assuming learning rate causes error here

5. You have a user-item rating matrix with many missing values. Which approach best uses matrix factorization to predict missing ratings?

hard

A. Fill missing values with zeros and factorize the full matrix directly

B. Ignore missing values and factorize only the submatrix of known ratings

C. Factorize the matrix into two smaller matrices and use gradient descent to minimize error only on known ratings

D. Use matrix addition to combine user and item matrices

Solution

1. Step 1: Understand handling missing values in matrix factorization

   Missing values should not be treated as zeros; only known ratings guide factorization.

2. Step 2: Use gradient descent minimizing error on known ratings

   This approach updates factor matrices to predict missing values accurately.

3. Final Answer:

   Factorize the matrix into two smaller matrices and use gradient descent to minimize error only on known ratings -> Option C

4. Quick Check:

   Minimize error on known ratings for prediction [OK]

Hint: Train only on known ratings, predict missing [OK]

Common Mistakes:

• Filling missing values with zeros distorts learning
• Ignoring missing values loses data context
• Using addition instead of multiplication for factorization