Regularization (Ridge, Lasso) in ML Python - ML Experiment: Train & Evaluate

Experiment - Regularization (Ridge, Lasso)

Problem:You want to predict house prices using many features. The current model fits training data very well but performs poorly on new data.

Current Metrics:Training R2 score: 0.95, Validation R2 score: 0.65

Issue:The model is overfitting: it learns training data too closely but fails to generalize to validation data.

Your Task

Reduce overfitting by applying Ridge or Lasso regularization to improve validation R2 score to at least 0.80 while keeping training R2 above 0.85.

Use only Ridge or Lasso regression models.

Do not change the dataset or feature set.

Keep the training process simple and explainable.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import load_boston
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

# Load dataset
X, y = load_boston(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Ridge regression with regularization
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
y_train_pred_ridge = ridge.predict(X_train)
y_val_pred_ridge = ridge.predict(X_val)
train_r2_ridge = r2_score(y_train, y_train_pred_ridge)
val_r2_ridge = r2_score(y_val, y_val_pred_ridge)

# Train Lasso regression with regularization
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)
y_train_pred_lasso = lasso.predict(X_train)
y_val_pred_lasso = lasso.predict(X_val)
train_r2_lasso = r2_score(y_train, y_train_pred_lasso)
val_r2_lasso = r2_score(y_val, y_val_pred_lasso)

print(f"Ridge Training R2: {train_r2_ridge:.2f}, Validation R2: {val_r2_ridge:.2f}")
print(f"Lasso Training R2: {train_r2_lasso:.2f}, Validation R2: {val_r2_lasso:.2f}")

Added Ridge and Lasso regression models with regularization.

Set alpha parameters to control regularization strength.

Evaluated both models on training and validation data.

Results Interpretation

Before regularization: Training R2 = 0.95, Validation R2 = 0.65 (overfitting)

After Ridge regularization: Training R2 = 0.90, Validation R2 = 0.82 (better generalization)

After Lasso regularization: Training R2 = 0.87, Validation R2 = 0.80 (better generalization with feature selection)

Regularization reduces overfitting by penalizing large coefficients, which helps the model generalize better to new data.

Bonus Experiment

Try using ElasticNet regularization, which combines Ridge and Lasso, and tune its parameters to improve validation performance further.

💡 Hint

Adjust both alpha and l1_ratio parameters to balance between Ridge and Lasso effects.

Practice

1. What is the main purpose of regularization in machine learning models?

easy

A. To prevent overfitting by adding a penalty to large weights

B. To increase the size of the training dataset

C. To speed up the training process by reducing data

D. To remove all features from the model

Solution

1. Step 1: Understand overfitting

   Overfitting happens when a model learns noise instead of the true pattern, causing poor performance on new data.

2. Step 2: Role of regularization

   Regularization adds a penalty to large weights, which helps the model generalize better by keeping weights small.

3. Final Answer:

   To prevent overfitting by adding a penalty to large weights -> Option A

4. Quick Check:

   Regularization = Prevent overfitting [OK]

Hint: Regularization controls model complexity by penalizing big weights [OK]

Common Mistakes:

• Thinking regularization increases data size
• Confusing regularization with feature removal
• Believing regularization speeds up training

2. Which of the following is the correct way to import Ridge regression from scikit-learn in Python?

easy

A. import sklearn.linear_model.Ridge

B. from sklearn.linear_model import Ridge

C. from sklearn import Ridge

D. import Ridge from sklearn.linear_model

Solution

1. Step 1: Recall Python import syntax

   In Python, to import a class from a module, the syntax is 'from module import Class'.

2. Step 2: Apply to Ridge regression

   Ridge is a class inside sklearn.linear_model, so the correct import is 'from sklearn.linear_model import Ridge'.

3. Final Answer:

   from sklearn.linear_model import Ridge -> Option B

4. Quick Check:

   Correct import syntax = from sklearn.linear_model import Ridge [OK]

Hint: Use 'from module import Class' to import classes in Python [OK]

Common Mistakes:

• Using 'import Class from module' which is invalid syntax
• Trying to import module and class together incorrectly
• Using dot notation in import statement

3. Consider the following Python code using Lasso regression:

from sklearn.linear_model import Lasso
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])
model = Lasso(alpha=2.0)
model.fit(X, y)
pred = model.predict(np.array([[7, 8]]))
print(round(pred[0], 2))

What is the printed output?

medium

A. 2.5

B. 2.0

C. 3.0

D. 1.0

Solution

1. Step 1: Understand Lasso fitting on small data

   Lasso applies L1 penalty, shrinking some coefficients possibly to zero. With alpha=2.0, coefficients shrink significantly.

2. Step 2: Predict value for input [7, 8]

   Given the linear trend in X and y (unregularized prediction 4.0), with alpha=2.0 the prediction is approximately 2.5 after regularization.

3. Final Answer:

   2.5 -> Option A

4. Quick Check:

   Lasso prediction ≈ 2.5 [OK]

Hint: Lasso shrinks coefficients, prediction near linear trend but reduced by alpha [OK]

Common Mistakes:

• Assuming prediction equals last y value
• Ignoring effect of alpha on coefficients
• Confusing Ridge and Lasso penalties

4. The following code tries to use Ridge regression but throws an error:

from sklearn.linear_model import Ridge
model = Ridge(alpha='one')
model.fit(X_train, y_train)

What is the error and how to fix it?

medium

A. SyntaxError due to wrong import statement; fix import syntax

B. NameError because Ridge is not imported; fix by importing Ridge

C. ValueError because X_train is missing; fix by defining X_train

D. ValueError because alpha must be a float; fix by using alpha=1.0

Solution

1. Step 1: Identify the error cause

   Alpha parameter expects a numeric value (float), but 'one' is a string causing ValueError.

2. Step 2: Fix the error

   Replace alpha='one' with alpha=1.0 to provide a valid float value.

3. Final Answer:

   ValueError because alpha must be a float; fix by using alpha=1.0 -> Option D

4. Quick Check:

   Alpha must be float, not string [OK]

Hint: Alpha must be a number, not a string [OK]

Common Mistakes:

• Passing string instead of float for alpha
• Ignoring error message details
• Assuming import error without checking code

5. You have a dataset with many features, some of which are irrelevant. You want a model that both reduces overfitting and selects important features by setting some coefficients exactly to zero. Which regularization method should you choose?

hard

A. Ridge regression because it shrinks coefficients but keeps all features

B. Neither Ridge nor Lasso; use standard linear regression

C. Lasso regression because it can set some coefficients to zero, removing features

D. Use Ridge with very small alpha to remove features

Solution

1. Step 1: Understand feature selection need

   To remove irrelevant features, the model must set some coefficients exactly to zero.

2. Step 2: Compare Ridge and Lasso

   Ridge shrinks coefficients but does not zero them out; Lasso uses L1 penalty and can zero coefficients, effectively selecting features.

3. Final Answer:

   Lasso regression because it can set some coefficients to zero, removing features -> Option C

4. Quick Check:

   Lasso = feature selection + regularization [OK]

Hint: Lasso zeros coefficients; Ridge only shrinks [OK]

Common Mistakes:

• Thinking Ridge removes features by zeroing coefficients
• Choosing no regularization for feature selection
• Confusing alpha size effect on feature removal