Regularization (Ridge, Lasso) in ML Python

Introduction

Regularization helps a model avoid overfitting by keeping it simple. It adds a small penalty to large model weights to make predictions more reliable on new data.

When your model fits the training data too well but performs poorly on new data.

When you have many features and want to reduce the impact of less important ones.

When you want to prevent your model from becoming too complex and sensitive to noise.

When you want to improve model stability and generalization.

When you want to select important features automatically (Lasso).

Syntax

ML Python

from sklearn.linear_model import Ridge, Lasso

ridge_model = Ridge(alpha=1.0)
lasso_model = Lasso(alpha=1.0)

ridge_model.fit(X_train, y_train)
lasso_model.fit(X_train, y_train)

ridge_predictions = ridge_model.predict(X_test)
lasso_predictions = lasso_model.predict(X_test)

alpha controls how strong the penalty is. Higher alpha means more regularization.

Ridge uses squared weights penalty (L2), Lasso uses absolute weights penalty (L1).

Examples

Creates a Ridge model with moderate regularization and fits it to training data.

ML Python

ridge = Ridge(alpha=0.5)
ridge.fit(X_train, y_train)

Creates a Lasso model with light regularization and fits it to training data.

ML Python

lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)

Predicts target values for test data using Ridge and prints first 5 predictions.

ML Python

ridge_predictions = ridge.predict(X_test)
print(ridge_predictions[:5])

Sample Model

This program creates a simple regression dataset, splits it, trains Ridge and Lasso models, then prints their mean squared errors and coefficients. You can see how regularization affects the model weights.

ML Python

from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Create sample data
X, y = make_regression(n_samples=100, n_features=5, noise=10, random_state=42)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create models
ridge = Ridge(alpha=1.0)
lasso = Lasso(alpha=1.0)

# Train models
ridge.fit(X_train, y_train)
lasso.fit(X_train, y_train)

# Predict
ridge_pred = ridge.predict(X_test)
lasso_pred = lasso.predict(X_test)

# Calculate errors
ridge_mse = mean_squared_error(y_test, ridge_pred)
lasso_mse = mean_squared_error(y_test, lasso_pred)

print(f"Ridge MSE: {ridge_mse:.2f}")
print(f"Lasso MSE: {lasso_mse:.2f}")
print(f"Ridge Coefficients: {ridge.coef_}")
print(f"Lasso Coefficients: {lasso.coef_}")

Important Notes

Ridge keeps all features but shrinks their weights.

Lasso can shrink some weights exactly to zero, effectively selecting features.

Choosing the right alpha is important; use cross-validation to find the best value.

Summary

Regularization helps prevent overfitting by adding a penalty to large weights.

Ridge uses L2 penalty and shrinks weights but keeps all features.

Lasso uses L1 penalty and can remove less important features by setting weights to zero.

Practice

1. What is the main purpose of regularization in machine learning models?

easy

A. To prevent overfitting by adding a penalty to large weights

B. To increase the size of the training dataset

C. To speed up the training process by reducing data

D. To remove all features from the model

Solution

1. Step 1: Understand overfitting

   Overfitting happens when a model learns noise instead of the true pattern, causing poor performance on new data.

2. Step 2: Role of regularization

   Regularization adds a penalty to large weights, which helps the model generalize better by keeping weights small.

3. Final Answer:

   To prevent overfitting by adding a penalty to large weights -> Option A

4. Quick Check:

   Regularization = Prevent overfitting [OK]

Hint: Regularization controls model complexity by penalizing big weights [OK]

Common Mistakes:

• Thinking regularization increases data size
• Confusing regularization with feature removal
• Believing regularization speeds up training

2. Which of the following is the correct way to import Ridge regression from scikit-learn in Python?

easy

A. import sklearn.linear_model.Ridge

B. from sklearn.linear_model import Ridge

C. from sklearn import Ridge

D. import Ridge from sklearn.linear_model

Solution

1. Step 1: Recall Python import syntax

   In Python, to import a class from a module, the syntax is 'from module import Class'.

2. Step 2: Apply to Ridge regression

   Ridge is a class inside sklearn.linear_model, so the correct import is 'from sklearn.linear_model import Ridge'.

3. Final Answer:

   from sklearn.linear_model import Ridge -> Option B

4. Quick Check:

   Correct import syntax = from sklearn.linear_model import Ridge [OK]

Hint: Use 'from module import Class' to import classes in Python [OK]

Common Mistakes:

• Using 'import Class from module' which is invalid syntax
• Trying to import module and class together incorrectly
• Using dot notation in import statement

3. Consider the following Python code using Lasso regression:

from sklearn.linear_model import Lasso
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])
model = Lasso(alpha=2.0)
model.fit(X, y)
pred = model.predict(np.array([[7, 8]]))
print(round(pred[0], 2))

What is the printed output?

medium

A. 2.5

B. 2.0

C. 3.0

D. 1.0

Solution

1. Step 1: Understand Lasso fitting on small data

   Lasso applies L1 penalty, shrinking some coefficients possibly to zero. With alpha=2.0, coefficients shrink significantly.

2. Step 2: Predict value for input [7, 8]

   Given the linear trend in X and y (unregularized prediction 4.0), with alpha=2.0 the prediction is approximately 2.5 after regularization.

3. Final Answer:

   2.5 -> Option A

4. Quick Check:

   Lasso prediction ≈ 2.5 [OK]

Hint: Lasso shrinks coefficients, prediction near linear trend but reduced by alpha [OK]

Common Mistakes:

• Assuming prediction equals last y value
• Ignoring effect of alpha on coefficients
• Confusing Ridge and Lasso penalties

4. The following code tries to use Ridge regression but throws an error:

from sklearn.linear_model import Ridge
model = Ridge(alpha='one')
model.fit(X_train, y_train)

What is the error and how to fix it?

medium

A. SyntaxError due to wrong import statement; fix import syntax

B. NameError because Ridge is not imported; fix by importing Ridge

C. ValueError because X_train is missing; fix by defining X_train

D. ValueError because alpha must be a float; fix by using alpha=1.0

Solution

1. Step 1: Identify the error cause

   Alpha parameter expects a numeric value (float), but 'one' is a string causing ValueError.

2. Step 2: Fix the error

   Replace alpha='one' with alpha=1.0 to provide a valid float value.

3. Final Answer:

   ValueError because alpha must be a float; fix by using alpha=1.0 -> Option D

4. Quick Check:

   Alpha must be float, not string [OK]

Hint: Alpha must be a number, not a string [OK]

Common Mistakes:

• Passing string instead of float for alpha
• Ignoring error message details
• Assuming import error without checking code

5. You have a dataset with many features, some of which are irrelevant. You want a model that both reduces overfitting and selects important features by setting some coefficients exactly to zero. Which regularization method should you choose?

hard

A. Ridge regression because it shrinks coefficients but keeps all features

B. Neither Ridge nor Lasso; use standard linear regression

C. Lasso regression because it can set some coefficients to zero, removing features

D. Use Ridge with very small alpha to remove features

Solution

1. Step 1: Understand feature selection need

   To remove irrelevant features, the model must set some coefficients exactly to zero.

2. Step 2: Compare Ridge and Lasso

   Ridge shrinks coefficients but does not zero them out; Lasso uses L1 penalty and can zero coefficients, effectively selecting features.

3. Final Answer:

   Lasso regression because it can set some coefficients to zero, removing features -> Option C

4. Quick Check:

   Lasso = feature selection + regularization [OK]

Hint: Lasso zeros coefficients; Ridge only shrinks [OK]

Common Mistakes:

• Thinking Ridge removes features by zeroing coefficients
• Choosing no regularization for feature selection
• Confusing alpha size effect on feature removal