Multiple linear regression in ML Python - ML Experiment: Train & Evaluate

Experiment - Multiple linear regression

Problem:Predict house prices using multiple features like size, number of bedrooms, and age of the house.

Current Metrics:Training R2 score: 0.95, Validation R2 score: 0.70

Issue:The model is overfitting: training score is very high but validation score is much lower.

Your Task

Reduce overfitting so that validation R2 score improves to at least 0.85 while keeping training R2 score below 0.90.

You can only modify the model training process and data preprocessing steps.

Do not change the dataset or add new features.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score

# Generate synthetic data simulating house features and prices
X, y = make_regression(n_samples=500, n_features=3, noise=15, random_state=42)

# Split data into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)

# Use Ridge regression with cross-validation to find best alpha
ridge = Ridge()
param_grid = {'alpha': [0.1, 1, 10, 100]}
grid_search = GridSearchCV(ridge, param_grid, cv=5)
grid_search.fit(X_train_scaled, y_train)

# Best model
best_ridge = grid_search.best_estimator_

# Predictions
train_pred = best_ridge.predict(X_train_scaled)
val_pred = best_ridge.predict(X_val_scaled)

# Calculate R2 scores
train_r2 = r2_score(y_train, train_pred)
val_r2 = r2_score(y_val, val_pred)

print(f"Training R2 score: {train_r2:.2f}")
print(f"Validation R2 score: {val_r2:.2f}")

Added feature scaling using StandardScaler to normalize input features.

Replaced simple linear regression with Ridge regression to add regularization.

Used GridSearchCV to find the best regularization strength (alpha).

Results Interpretation

Before: Training R2 = 0.95, Validation R2 = 0.70 (overfitting)

After: Training R2 = 0.88, Validation R2 = 0.86 (better generalization)

Adding regularization and scaling features helps reduce overfitting, improving the model's ability to predict new data accurately.

Bonus Experiment

Try using Lasso regression instead of Ridge to see how feature selection affects overfitting and model performance.

💡 Hint

Lasso can shrink some coefficients to zero, effectively selecting important features and possibly improving validation score.

Practice

1. What does multiple linear regression do in simple terms?

easy

A. It predicts a result using several input features combined.

B. It groups data into clusters based on similarity.

C. It reduces the number of features by combining them.

D. It finds the shortest path between two points.

Solution

1. Step 1: Understand the purpose of multiple linear regression

   Multiple linear regression uses multiple inputs to predict a single output by finding the best weights for each input.

2. Step 2: Compare options with this understanding

   Only It predicts a result using several input features combined. correctly describes predicting a result using several inputs combined.

3. Final Answer:

   It predicts a result using several input features combined. -> Option A

4. Quick Check:

   Multiple inputs predict output = A [OK]

Hint: Remember: multiple inputs, one output prediction [OK]

Common Mistakes:

• Confusing regression with clustering
• Thinking it reduces features instead of weighting them
• Mixing it up with pathfinding algorithms

2. Which of the following is the correct formula for multiple linear regression prediction?

easy

A. y = w0 + w1*x1 + w2*x2 + ... + wn*xn

B. y = w0 * x1 + w1 * x2 + ... + wn * xn

C. y = w0 / x1 + w1 / x2 + ... + wn / xn

D. y = w0 - w1*x1 - w2*x2 - ... - wn*xn

Solution

1. Step 1: Recall the multiple linear regression formula

   The prediction is a sum of a bias term plus each input multiplied by its weight: y = w0 + w1*x1 + w2*x2 + ... + wn*xn.

2. Step 2: Check each option against the formula

   Only y = w0 + w1*x1 + w2*x2 + ... + wn*xn matches the correct formula with addition and multiplication.

3. Final Answer:

   y = w0 + w1*x1 + w2*x2 + ... + wn*xn -> Option A

4. Quick Check:

   Sum of weighted inputs plus bias = B [OK]

Hint: Sum weighted inputs plus bias for prediction [OK]

Common Mistakes:

• Using multiplication instead of addition between terms
• Dividing weights by inputs
• Subtracting weights instead of adding

3. Given weights w0=2, w1=3, w2=-1 and inputs x1=4, x2=5, what is the predicted value y using multiple linear regression?

medium

A. 9.5

B. 9

C. 5

D. 10

Solution

1. Step 1: Apply the formula y = w0 + w1*x1 + w2*x2

   Substitute values: y = 2 + 3*4 + (-1)*5.

2. Step 2: Calculate the prediction

   Calculate: 3*4 = 12, (-1)*5 = -5, sum = 2 + 12 - 5 = 9.

3. Step 3: Recheck calculation

   Sum is 9, but 10 says 10, so re-check: 2 + 12 - 5 = 9, so correct answer is 9.

4. Final Answer:

   9 -> Option B

5. Quick Check:

   2 + 12 - 5 = 9 [OK]

Hint: Multiply weights by inputs, then add bias [OK]

Common Mistakes:

• Adding weights instead of multiplying by inputs
• Forgetting to add bias term w0
• Mixing up signs of weights

4. Identify the error in this Python code for multiple linear regression prediction:

def predict(x1, x2, w0, w1, w2):
    return w0 + w1 * x1 + w2 + x2

print(predict(3, 4, 1, 2, 3))

medium

A. The print statement syntax is incorrect.

B. The function is missing a return statement.

C. The function parameters are in wrong order.

D. The term w2 + x2 should be w2 * x2 for correct multiplication.

Solution

1. Step 1: Analyze the formula inside the function

   The formula should multiply each input by its weight: w0 + w1*x1 + w2*x2.

2. Step 2: Identify the mistake in the code

   The code uses w2 + x2 instead of w2 * x2, which adds instead of multiplies.

3. Final Answer:

   The term w2 + x2 should be w2 * x2 for correct multiplication. -> Option D

4. Quick Check:

   Multiplication needed, not addition = A [OK]

Hint: Check all input-weight pairs use multiplication [OK]

Common Mistakes:

• Using + instead of * between weight and input
• Forgetting return statement (not here though)
• Mixing parameter order
• Incorrect print syntax

5. You have a dataset with features: hours studied (x1), hours slept (x2), and exam score (y). You fit a multiple linear regression model and get weights: w0=5, w1=4, w2=2. What does the weight w1=4 tell you about hours studied?

hard

A. Hours slept affects exam score more than hours studied.

B. Hours studied has no effect on exam score.

C. For each extra hour studied, exam score increases by 4 points, holding sleep constant.

D. Exam score decreases by 4 points for each hour studied.

Solution

1. Step 1: Understand the meaning of weights in multiple linear regression

   Each weight shows how much the output changes when that input increases by one unit, keeping others fixed.

2. Step 2: Interpret w1=4 for hours studied

   A weight of 4 means exam score increases by 4 points for each additional hour studied, assuming hours slept stays the same.

3. Final Answer:

   For each extra hour studied, exam score increases by 4 points, holding sleep constant. -> Option C

4. Quick Check:

   Weight shows effect per unit input = D [OK]

Hint: Weight = change in output per unit input [OK]

Common Mistakes:

• Thinking weight means no effect
• Confusing positive and negative effect
• Mixing effects of different features