Bird
Raised Fist0

Overfitting and Underfitting - Diagnosing and Fixing

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
🐛
The Feature Importance Anomaly
Started after the last feature update two weeks ago.

A recommendation system model predicting user engagement for a streaming service with 5M daily active users.

Symptom
Model accuracy is stable at 0.85, but recall has dropped to 0.0 for a key user segment.
📡
Signals
metric
Recall for users aged 18-24
Recall dropped from 0.76 to 0.0 after the update
feature_importance
Feature importance for 'user_age_group'
Importance dropped from 0.22 to 0.01 post-update
distribution
Distribution of 'user_age_group' in training vs. live data
Live data has 30% users aged 18-24, training data has 5%
log
Model training logs
No anomalies, but logs show 'user_age_group' was not used in recent training
metric
Overall model accuracy
Stable at 0.85
🔬
Hypotheses
The model is underfitting due to a lack of representation of the 18-24 age group in training data.
distribution
Verify: Check the distribution of 'user_age_group' in the training dataset.
import pandas as pd
train_data = pd.read_csv('train_data.csv')
train_data['user_age_group'].value_counts(normalize=True)
The feature 'user_age_group' was mistakenly dropped during the last model update.
feature_importancelog
Verify: Review the feature engineering pipeline to ensure 'user_age_group' is included.
features_used = ['user_age_group', 'other_features']
assert 'user_age_group' in features_used
🎯
Root Cause
The feature 'user_age_group' was inadvertently excluded from the model during the last update.

A change in the feature engineering script led to 'user_age_group' being dropped.

🔧
Fix Strategy
ImmediateRollback to the previous model version that included 'user_age_group'.
Short-termUpdate the feature engineering script to ensure 'user_age_group' is included.
Long-termImplement a feature inclusion checklist in the deployment pipeline.
🐍
Python Fix
Python
features = ['user_age_group', 'other_features']
model.fit(train_data[features], train_data['target'])
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to too many features.
Why candidates think this: Feature importance changes suggest feature issues.
Why wrong: The issue is not too many features but a critical feature being dropped.
The data pipeline is corrupt, causing data leakage.
Why candidates think this: The drastic drop in recall suggests data issues.
Why wrong: Data leakage would affect more metrics, not just recall for one segment.
📊
Monitoring to Add
  • Monitor feature importance changes post-deployment.
  • Set up alerts for drastic drops in recall for key segments.
  • Track distribution shifts between training and live data.
📋
Process Change

Introduce a feature review step before model deployment.

🏗️
Architectural Change

Automate feature selection validation to prevent accidental exclusions.

💡
Key Insight
Feature exclusion can mimic underfitting symptoms, especially when critical features are dropped.

Practice

(1/5)
1. In the context of dropout regularization, what happens to the dropout rate during inference?
easy
A. The dropout rate is decreased to zero.
B. The dropout rate is increased to improve generalization.
C. The dropout rate remains the same as during training.
D. The dropout rate is adjusted based on validation performance.

Solution

  1. Step 1: Understand dropout behavior during inference

    During inference, dropout is not applied, meaning the dropout rate is effectively zero.
  2. Step 2: Eliminate wrong options

    The dropout rate is increased to improve generalization. is incorrect because increasing dropout during inference would not improve generalization. The dropout rate remains the same as during training. is incorrect because dropout is not applied during inference. The dropout rate is adjusted based on validation performance. is incorrect because dropout rate is not adjusted based on validation performance during inference.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Dropout off during inference. [OK]
Hint: Dropout off during inference. [OK]
Common Mistakes:
2. In the context of Graph Neural Networks, which component is primarily responsible for aggregating information from neighboring nodes?
easy
A. Dropout layer
B. Activation function
C. Message passing mechanism
D. Batch normalization

Solution

  1. Step 1: Identify the role of each component

    The message passing mechanism is designed to aggregate information from neighboring nodes in a graph.
  2. Step 2: Eliminate wrong options

    Activation functions, dropout layers, and batch normalization do not specifically aggregate information from neighboring nodes.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Message passing aggregates neighbor info in GNNs. [check]
Hint: Message passing aggregates neighbor info in GNNs. [check]
Common Mistakes:
3. What is the output of the following Python code snippet using NumPy for logistic regression log-odds calculation?
import numpy as np
coefficients = np.array([0.5, -0.25])
features = np.array([2, 3])
log_odds = np.dot(coefficients, features)
print(log_odds)
easy
A. -0.25
B. 0.25
C. 1.25
D. 2.75

Solution

  1. Step 1: Calculate the dot product.

    The dot product is calculated as (0.5 * 2) + (-0.25 * 3) = 1.0 - 0.75 = 0.25.
  2. Step 2: Identify the correct output.

    -0.25 is correct because the calculation results in 0.25. 1.25 is incorrect as it suggests a miscalculation. 0.25 is incorrect due to a sign error. 2.75 is incorrect due to a misunderstanding of dot product.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Dot product = sum of products [OK]
Hint: Dot product = sum of products [OK]
Common Mistakes:
4.
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 1, 0, 1])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)

print(X_train.shape, X_test.shape)
What is the output of this code snippet?
easy
A. (2, 2) (2, 1)
B. (3, 2) (1, 2)
C. (2, 2) (2, 2)
D. (1, 2) (3, 2)

Solution

  1. Step 1: Understand train_test_split behavior.

    With test_size=0.5, the data is split equally into train and test sets.
  2. Step 2: Calculate expected shapes.

    Original data has 4 samples, so both train and test sets will have 2 samples each.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Equal split with test_size=0.5. [OK]
Hint: Equal split with test_size=0.5. [OK]
Common Mistakes:
5. You are choosing between a decision tree and a linear regression model for a dataset with a complex, non-linear relationship. Which model should you choose and why?
medium
A. Linear regression, because it is less likely to overfit.
B. Linear regression, because it is computationally cheaper.
C. Decision tree, because it can capture non-linear relationships.
D. Decision tree, because it has lower bias.

Solution

  1. Step 1: Analyze the dataset characteristics

    The dataset has a complex, non-linear relationship.
  2. Step 2: Choose the appropriate model

    Decision trees can capture non-linear relationships, making them suitable for this dataset.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    Decision trees capture non-linear patterns. [OK]
Hint: Decision trees capture non-linear patterns. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard