Regularization (Ridge, Lasso) in ML Python - Model Pipeline Trace

Model Pipeline - Regularization (Ridge, Lasso)

This pipeline shows how regularization techniques like Ridge and Lasso help a linear regression model avoid overfitting by adding penalties to the model's complexity. It improves prediction on new data by controlling the size of the coefficients.

Data Flow - 6 Stages

1Data in

1000 rows x 10 columns→Raw dataset with 10 features and target variable→1000 rows x 10 columns

Feature1=5.1, Feature2=3.5, ..., Feature10=0.2, Target=15.3

↓

2Preprocessing

1000 rows x 10 columns→Standardize features to zero mean and unit variance→1000 rows x 10 columns

Feature1=0.12, Feature2=-0.45, ..., Feature10=1.03

↓

3Feature Engineering

1000 rows x 10 columns→No new features added, features ready for model→1000 rows x 10 columns

Standardized features as above

↓

4Model Trains

800 rows x 10 columns→Train Ridge or Lasso regression on training set→Model with 10 coefficients

Coefficients: [0.5, 0.0, 0.3, ..., 0.1]

↓

5Metrics Improve

200 rows x 10 columns→Evaluate model on test set, measure loss and R2 score→Loss=0.25, R2=0.85

Predicted vs actual target values close

↓

6Prediction

1 row x 10 columns→Model predicts target for new data→1 predicted value

Input features standardized: [0.1, -0.2, ..., 0.5], Prediction=0.28

Training Trace - Epoch by Epoch

Loss
1.2 |*       
1.0 | *      
0.8 |  *     
0.6 |   *    
0.4 |    *   
0.2 |     *  
0.0 +--------
      1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.20	N/A	Initial loss high, model coefficients large
2	0.85	N/A	Loss decreased as coefficients shrink due to regularization
3	0.60	N/A	Model starts to generalize better, coefficients smaller
4	0.40	N/A	Loss continues to decrease, less overfitting
5	0.30	N/A	Loss stabilizes, model well-regularized

Prediction Trace - 3 Layers

Layer 1: Input features

Layer 2: Linear combination with coefficients

Layer 3: Add intercept

Model Quiz - 3 Questions

What is the main purpose of regularization in this pipeline?

A To make the model run faster

B To reduce model complexity and prevent overfitting

C To increase the number of features

D To increase training loss

Key Insight

Regularization techniques like Ridge and Lasso add penalties to the size of coefficients. This helps the model avoid fitting noise in the training data, leading to better predictions on new data by keeping the model simpler and more general.

Practice

1. What is the main purpose of regularization in machine learning models?

easy

A. To prevent overfitting by adding a penalty to large weights

B. To increase the size of the training dataset

C. To speed up the training process by reducing data

D. To remove all features from the model

Solution

1. Step 1: Understand overfitting

   Overfitting happens when a model learns noise instead of the true pattern, causing poor performance on new data.

2. Step 2: Role of regularization

   Regularization adds a penalty to large weights, which helps the model generalize better by keeping weights small.

3. Final Answer:

   To prevent overfitting by adding a penalty to large weights -> Option A

4. Quick Check:

   Regularization = Prevent overfitting [OK]

Hint: Regularization controls model complexity by penalizing big weights [OK]

Common Mistakes:

• Thinking regularization increases data size
• Confusing regularization with feature removal
• Believing regularization speeds up training

2. Which of the following is the correct way to import Ridge regression from scikit-learn in Python?

easy

A. import sklearn.linear_model.Ridge

B. from sklearn.linear_model import Ridge

C. from sklearn import Ridge

D. import Ridge from sklearn.linear_model

Solution

1. Step 1: Recall Python import syntax

   In Python, to import a class from a module, the syntax is 'from module import Class'.

2. Step 2: Apply to Ridge regression

   Ridge is a class inside sklearn.linear_model, so the correct import is 'from sklearn.linear_model import Ridge'.

3. Final Answer:

   from sklearn.linear_model import Ridge -> Option B

4. Quick Check:

   Correct import syntax = from sklearn.linear_model import Ridge [OK]

Hint: Use 'from module import Class' to import classes in Python [OK]

Common Mistakes:

• Using 'import Class from module' which is invalid syntax
• Trying to import module and class together incorrectly
• Using dot notation in import statement

3. Consider the following Python code using Lasso regression:

from sklearn.linear_model import Lasso
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])
model = Lasso(alpha=2.0)
model.fit(X, y)
pred = model.predict(np.array([[7, 8]]))
print(round(pred[0], 2))

What is the printed output?

medium

A. 2.5

B. 2.0

C. 3.0

D. 1.0

Solution

1. Step 1: Understand Lasso fitting on small data

   Lasso applies L1 penalty, shrinking some coefficients possibly to zero. With alpha=2.0, coefficients shrink significantly.

2. Step 2: Predict value for input [7, 8]

   Given the linear trend in X and y (unregularized prediction 4.0), with alpha=2.0 the prediction is approximately 2.5 after regularization.

3. Final Answer:

   2.5 -> Option A

4. Quick Check:

   Lasso prediction ≈ 2.5 [OK]

Hint: Lasso shrinks coefficients, prediction near linear trend but reduced by alpha [OK]

Common Mistakes:

• Assuming prediction equals last y value
• Ignoring effect of alpha on coefficients
• Confusing Ridge and Lasso penalties

4. The following code tries to use Ridge regression but throws an error:

from sklearn.linear_model import Ridge
model = Ridge(alpha='one')
model.fit(X_train, y_train)

What is the error and how to fix it?

medium

A. SyntaxError due to wrong import statement; fix import syntax

B. NameError because Ridge is not imported; fix by importing Ridge

C. ValueError because X_train is missing; fix by defining X_train

D. ValueError because alpha must be a float; fix by using alpha=1.0

Solution

1. Step 1: Identify the error cause

   Alpha parameter expects a numeric value (float), but 'one' is a string causing ValueError.

2. Step 2: Fix the error

   Replace alpha='one' with alpha=1.0 to provide a valid float value.

3. Final Answer:

   ValueError because alpha must be a float; fix by using alpha=1.0 -> Option D

4. Quick Check:

   Alpha must be float, not string [OK]

Hint: Alpha must be a number, not a string [OK]

Common Mistakes:

• Passing string instead of float for alpha
• Ignoring error message details
• Assuming import error without checking code

5. You have a dataset with many features, some of which are irrelevant. You want a model that both reduces overfitting and selects important features by setting some coefficients exactly to zero. Which regularization method should you choose?

hard

A. Ridge regression because it shrinks coefficients but keeps all features

B. Neither Ridge nor Lasso; use standard linear regression

C. Lasso regression because it can set some coefficients to zero, removing features

D. Use Ridge with very small alpha to remove features

Solution

1. Step 1: Understand feature selection need

   To remove irrelevant features, the model must set some coefficients exactly to zero.

2. Step 2: Compare Ridge and Lasso

   Ridge shrinks coefficients but does not zero them out; Lasso uses L1 penalty and can zero coefficients, effectively selecting features.

3. Final Answer:

   Lasso regression because it can set some coefficients to zero, removing features -> Option C

4. Quick Check:

   Lasso = feature selection + regularization [OK]

Hint: Lasso zeros coefficients; Ridge only shrinks [OK]

Common Mistakes:

• Thinking Ridge removes features by zeroing coefficients
• Choosing no regularization for feature selection
• Confusing alpha size effect on feature removal