Bird
Raised Fist0

Regularization - L1 Lasso, L2 Ridge, Elastic Net, Why Each Works

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started after the last model update two weeks ago.

A credit scoring model predicting default risk for a bank's loan applications, processing 500,000 applications per month.

Symptom
Model accuracy remains stable, but recall for the minority class (defaults) has dropped to 0.
📡
Signals
metric
Recall for the minority class (defaults) in the test set.
Recall dropped from 0.65 to 0.00 after the last update.
feature_importance
Feature importance ranking from the model.
Feature 'debt_to_income_ratio' importance dropped from 0.22 to 0.01.
distribution
Distribution of predicted probabilities for the minority class.
Predicted probabilities for defaults are clustered near 0.
log
Model training logs showing regularization parameters.
L1 regularization parameter increased from 0.1 to 1.0.
metric
Overall model accuracy on the test set.
Accuracy remained stable at 0.85.
🔬
Hypotheses
The increase in L1 regularization is causing important features to be zeroed out.
Supporting Signals
Feature 'debt_to_income_ratio' importance dropped from 0.22 to 0.01.L1 regularization parameter increased from 0.1 to 1.0.
Verify: Check the coefficients of the model to see if important features have been zeroed out.
import numpy as np
coefficients = np.array(model.coef_)
zeroed_features = np.sum(coefficients == 0)
print(f'Zeroed features: {zeroed_features}')
The model is overfitting to the majority class due to imbalanced data.
Supporting Signals
Recall dropped from 0.65 to 0.00 after the last update.Predicted probabilities for defaults are clustered near 0.
Verify: Check the class distribution in the training data and model predictions.
from collections import Counter
train_class_distribution = Counter(y_train)
pred_class_distribution = Counter(model.predict(X_test))
print(train_class_distribution, pred_class_distribution)
🎯
Root Cause
The increase in L1 regularization caused important features to be zeroed out, leading to poor recall for the minority class.
Why It Happened

A hyperparameter tuning process inadvertently increased the L1 regularization strength too much.

🔧
Fix Strategy
ImmediateRollback to the previous model version with lower L1 regularization.
Short-termRe-tune the regularization parameters with a focus on maintaining feature importance for critical features.
Long-termImplement a more robust hyperparameter tuning process that includes checks for class-specific metrics like recall.
🐍
Python Fix
Python
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(penalty='l1', C=0.1, solver='liblinear')
model.fit(X_train, y_train)
✗
Common Wrong Diagnoses
▼ Show
The model is simply overfitting to the training data.
Why candidates think this: Overfitting is a common issue when recall drops.
Why wrong: The stable accuracy and specific drop in recall indicate a regularization issue, not general overfitting.
The data pipeline introduced a bug affecting feature values.
Why candidates think this: Feature importance changes suggest data issues.
Why wrong: The logs show a clear change in regularization parameters, not data values.
📊
Monitoring to Add
  • Monitor recall for minority classes in real-time.
  • Track changes in feature importance after each model update.
  • Set alerts for drastic changes in regularization parameters.
📋
Process Change

Include class-specific metrics in the model evaluation criteria during hyperparameter tuning.

🏗️
Architectural Change

Adopt a model evaluation framework that automatically tests for class imbalance effects.

💡
Key Insight
Excessive L1 regularization can zero out critical features, drastically affecting recall for minority classes.

Practice

(1/5)
1. A neural network using ReLU activation is not learning. What could be a likely cause?
medium
A. The network suffers from the dying ReLU problem.
B. The network is too deep, causing vanishing gradients.
C. The batch size is too small.
D. The learning rate is too high.

Solution

  1. Step 1: Identify ReLU-specific issues.

    ReLU can suffer from the dying ReLU problem where neurons output zero for all inputs.

  2. Step 2: Evaluate each option.

    High learning rate (A) and small batch size (D) are not ReLU-specific. Vanishing gradients (B) are less likely with ReLU.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dying ReLU: neurons stuck at zero. [OK]
Hint: Dying ReLU: neurons stuck at zero. [OK]
Common Mistakes:
2. A model trained on a dataset with engineered features is underperforming on the validation set. What is the most likely cause?
medium
A. The model is too simple for the complexity of the features.
B. The features are over-engineered, causing overfitting.
C. The model has too many hyperparameters to tune.
D. The features are not normalized, leading to poor model performance.

Solution

  1. Step 1: Analyze the symptoms of underperformance.

    Underperformance on validation suggests overfitting or poor generalization.

  2. Step 2: Evaluate the role of feature engineering.

    Over-engineered features (C) can lead to overfitting, especially if they capture noise. Normalization (B) affects convergence but not generalization directly. Model complexity (A, D) is less likely the issue if features are the focus.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Over-engineered features often lead to overfitting. [OK]
Hint: Over-engineered features often lead to overfitting. [OK]
Common Mistakes:
3. When choosing between online and offline data collection for a recommendation system, which factor is most critical for deciding to use online data collection?
medium
A. The need for real-time user feedback integration.
B. The availability of large historical datasets.
C. The requirement for high computational resources.
D. The simplicity of the data processing pipeline.

Solution

  1. Step 1: Identify the key factor for online data collection

    Online data collection is crucial when real-time feedback is needed to update recommendations dynamically.

  2. Step 2: Eliminate factors not specific to online collection

    Options B, C, and D do not specifically require online data collection; they can be addressed with offline methods.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs = online collection. [OK]
Hint: Real-time needs = online collection. [OK]
Common Mistakes:
4. You are tasked with designing a GNN for a dynamic graph where nodes and edges change over time. Which approach would best handle this scenario?
hard
A. Adopt a dynamic GNN that updates with graph changes.
B. Implement a GAT with a fixed adjacency matrix.
C. Use a standard RNN to capture temporal changes.
D. Use a static GCN and retrain the model periodically.

Solution

  1. Step 1: Understand the graph dynamics

    Dynamic graphs require models that can adapt to changes in nodes and edges.

  2. Step 2: Evaluate each approach

    Static GCNs and fixed adjacency matrices cannot adapt to changes, and RNNs are not designed for graph structures.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dynamic GNNs adapt to graph changes. [check]
Hint: Dynamic GNNs adapt to graph changes. [check]
Common Mistakes:
5. You are tasked with debugging a production model that suddenly started underperforming. Which approach is most effective in systematically diagnosing the issue?
hard
A. Check for recent changes in data preprocessing
B. Immediately retrain the model with more data
C. Increase the model's complexity
D. Reduce the learning rate and retrain

Solution

  1. Step 1: Identify potential causes

    Sudden performance drops often relate to changes in data or environment.

  2. Step 2: Choose a systematic approach

    Checking for changes in data preprocessing is a logical first step to identify if the input data has changed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sudden drop = check data changes [OK]
Hint: Sudden drop = check data changes [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard