Regularization (Ridge, Lasso) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Regularization (Ridge, Lasso)

Which metric matters for Regularization and WHY

When using regularization like Ridge or Lasso, the key metric to watch is validation error (such as Mean Squared Error on validation data). This is because regularization helps prevent overfitting, so we want to see if the model performs well on new data, not just training data.

We also look at model complexity indirectly through the size of coefficients: Lasso can shrink some coefficients to zero, effectively selecting features, while Ridge shrinks coefficients but keeps them all.

In classification tasks, metrics like accuracy, precision, and recall on validation data are important to check if regularization improves generalization.

Confusion Matrix Example (Classification with Regularization)

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20    
      Negative           |    15    |   85    

      Total samples = 200
    

From this matrix, we calculate:

• Precision = 80 / (80 + 15) = 0.842
• Recall = 80 / (80 + 20) = 0.8
• Accuracy = (80 + 85) / 200 = 0.825

Regularization helps keep these metrics stable by avoiding overfitting to training data.

Precision vs Recall Tradeoff with Regularization

Regularization controls model complexity. If too strong, the model underfits and misses patterns, lowering both precision and recall.

If too weak, the model overfits, memorizing training data noise, which can increase false positives or false negatives.

For example, in spam detection:

• High precision means few good emails marked as spam (important to avoid losing important messages).
• High recall means catching most spam emails.

Regularization helps find a balance by simplifying the model to generalize well.

What Good vs Bad Metrics Look Like with Regularization

Good:

• Validation error close to training error (shows no overfitting)
• Stable coefficients (not too large or wildly changing)
• Balanced precision and recall on validation data

Bad:

• Very low training error but high validation error (overfitting)
• Coefficients very large or zeroed out too much (underfitting)
• Precision or recall very low, indicating poor generalization

Common Pitfalls in Regularization Metrics

• Ignoring validation error: Only looking at training error can hide overfitting.
• Data leakage: Using test data during training can falsely improve metrics.
• Over-regularization: Too strong regularization can cause underfitting, hurting performance.
• Not tuning regularization strength: Using default values without tuning can miss the best balance.
• Confusing precision and recall: Know which metric matters for your problem.

Self-Check Question

Your model with Lasso regularization has 98% training accuracy but only 12% recall on fraud cases in validation. Is this good for production?

Answer: No, this is not good. The model misses most fraud cases (low recall), which is critical in fraud detection. Despite high training accuracy, the model likely overfits or is too simple. You should adjust regularization strength or try other methods to improve recall.

Key Result

Regularization balances model complexity to reduce overfitting, improving validation error and generalization metrics like precision and recall.

Practice

1. What is the main purpose of regularization in machine learning models?

easy

A. To prevent overfitting by adding a penalty to large weights

B. To increase the size of the training dataset

C. To speed up the training process by reducing data

D. To remove all features from the model

Solution

1. Step 1: Understand overfitting

   Overfitting happens when a model learns noise instead of the true pattern, causing poor performance on new data.

2. Step 2: Role of regularization

   Regularization adds a penalty to large weights, which helps the model generalize better by keeping weights small.

3. Final Answer:

   To prevent overfitting by adding a penalty to large weights -> Option A

4. Quick Check:

   Regularization = Prevent overfitting [OK]

Hint: Regularization controls model complexity by penalizing big weights [OK]

Common Mistakes:

• Thinking regularization increases data size
• Confusing regularization with feature removal
• Believing regularization speeds up training

2. Which of the following is the correct way to import Ridge regression from scikit-learn in Python?

easy

A. import sklearn.linear_model.Ridge

B. from sklearn.linear_model import Ridge

C. from sklearn import Ridge

D. import Ridge from sklearn.linear_model

Solution

1. Step 1: Recall Python import syntax

   In Python, to import a class from a module, the syntax is 'from module import Class'.

2. Step 2: Apply to Ridge regression

   Ridge is a class inside sklearn.linear_model, so the correct import is 'from sklearn.linear_model import Ridge'.

3. Final Answer:

   from sklearn.linear_model import Ridge -> Option B

4. Quick Check:

   Correct import syntax = from sklearn.linear_model import Ridge [OK]

Hint: Use 'from module import Class' to import classes in Python [OK]

Common Mistakes:

• Using 'import Class from module' which is invalid syntax
• Trying to import module and class together incorrectly
• Using dot notation in import statement

3. Consider the following Python code using Lasso regression:

from sklearn.linear_model import Lasso
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])
model = Lasso(alpha=2.0)
model.fit(X, y)
pred = model.predict(np.array([[7, 8]]))
print(round(pred[0], 2))

What is the printed output?

medium

A. 2.5

B. 2.0

C. 3.0

D. 1.0

Solution

1. Step 1: Understand Lasso fitting on small data

   Lasso applies L1 penalty, shrinking some coefficients possibly to zero. With alpha=2.0, coefficients shrink significantly.

2. Step 2: Predict value for input [7, 8]

   Given the linear trend in X and y (unregularized prediction 4.0), with alpha=2.0 the prediction is approximately 2.5 after regularization.

3. Final Answer:

   2.5 -> Option A

4. Quick Check:

   Lasso prediction ≈ 2.5 [OK]

Hint: Lasso shrinks coefficients, prediction near linear trend but reduced by alpha [OK]

Common Mistakes:

• Assuming prediction equals last y value
• Ignoring effect of alpha on coefficients
• Confusing Ridge and Lasso penalties

4. The following code tries to use Ridge regression but throws an error:

from sklearn.linear_model import Ridge
model = Ridge(alpha='one')
model.fit(X_train, y_train)

What is the error and how to fix it?

medium

A. SyntaxError due to wrong import statement; fix import syntax

B. NameError because Ridge is not imported; fix by importing Ridge

C. ValueError because X_train is missing; fix by defining X_train

D. ValueError because alpha must be a float; fix by using alpha=1.0

Solution

1. Step 1: Identify the error cause

   Alpha parameter expects a numeric value (float), but 'one' is a string causing ValueError.

2. Step 2: Fix the error

   Replace alpha='one' with alpha=1.0 to provide a valid float value.

3. Final Answer:

   ValueError because alpha must be a float; fix by using alpha=1.0 -> Option D

4. Quick Check:

   Alpha must be float, not string [OK]

Hint: Alpha must be a number, not a string [OK]

Common Mistakes:

• Passing string instead of float for alpha
• Ignoring error message details
• Assuming import error without checking code

5. You have a dataset with many features, some of which are irrelevant. You want a model that both reduces overfitting and selects important features by setting some coefficients exactly to zero. Which regularization method should you choose?

hard

A. Ridge regression because it shrinks coefficients but keeps all features

B. Neither Ridge nor Lasso; use standard linear regression

C. Lasso regression because it can set some coefficients to zero, removing features

D. Use Ridge with very small alpha to remove features

Solution

1. Step 1: Understand feature selection need

   To remove irrelevant features, the model must set some coefficients exactly to zero.

2. Step 2: Compare Ridge and Lasso

   Ridge shrinks coefficients but does not zero them out; Lasso uses L1 penalty and can zero coefficients, effectively selecting features.

3. Final Answer:

   Lasso regression because it can set some coefficients to zero, removing features -> Option C

4. Quick Check:

   Lasso = feature selection + regularization [OK]

Hint: Lasso zeros coefficients; Ridge only shrinks [OK]

Common Mistakes:

• Thinking Ridge removes features by zeroing coefficients
• Choosing no regularization for feature selection
• Confusing alpha size effect on feature removal