Bird
Raised Fist0

Overfitting and Underfitting - Diagnosing and Fixing

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
๐ŸŽฏ
Diagnosing and Fixing Overfitting and Underfitting in a Predictive Model
40 min ยท junior
junior

You are part of a team of 3 ML engineers tasked with improving a predictive model for a financial application. The model predicts credit risk based on customer data. You have access to a dataset of 100,000 samples with 50 features. The model must be deployed in a production environment with a latency SLA of 100ms. Your team has 3 months to deliver a solution.

Given Info
  • The current model is a decision tree with a depth of 10.
  • The model's training accuracy is 95%, but validation accuracy is only 70%.
  • The dataset is imbalanced with 80% low-risk and 20% high-risk samples.
  • Feature engineering is limited to basic transformations due to time constraints.
๐Ÿ“‹
Guided Steps
1/5
Step 1
Identifying the Problem
What specific issue is the current model facing based on the given information?
๐Ÿ’ก Hint
Consider the discrepancy between training and validation accuracy.
Key Decision
Identifying overfitting due to high training accuracy and low validation accuracy.
Interviewer Expectation
Mid: Recognizes overfitting from accuracy discrepancy. Senior: Also considers the impact of dataset imbalance on model performance.
โš ๏ธCommon Wrong Answer
Assuming the model is underfitting due to low validation accuracy without considering training accuracy.
Step 2
Choosing a Model Adjustment
What changes can be made to the model to address overfitting?
๐Ÿ’ก Hint
Think about model complexity and regularization techniques.
Key Decision
Implementing regularization or reducing model complexity to prevent overfitting.
Interviewer Expectation
Mid: Suggests reducing tree depth or adding regularization. Senior: Proposes specific regularization techniques like pruning or L2 regularization and justifies choice based on constraints.
โš ๏ธCommon Wrong Answer
Increasing model complexity to improve validation accuracy without addressing overfitting.
Step 3
Handling Imbalanced Data
How can you address the imbalance in the dataset to improve model performance?
๐Ÿ’ก Hint
Consider techniques that adjust for class imbalance.
Key Decision
Using techniques like resampling or class weighting to handle imbalance.
Interviewer Expectation
Mid: Proposes basic resampling techniques. Senior: Evaluates pros and cons of resampling vs. class weighting and considers impact on latency.
โš ๏ธCommon Wrong Answer
Ignoring class imbalance, assuming it will not affect model performance.
Step 4
Evaluating Model Performance
What metrics should be used to evaluate the improved model?
๐Ÿ’ก Hint
Consider both offline and online metrics relevant to the application.
Key Decision
Using precision-recall metrics for imbalanced data and latency for production constraints.
Interviewer Expectation
Mid: Mentions precision and recall. Senior: Explains why precision-recall is preferred over accuracy for imbalanced data and discusses latency implications.
โš ๏ธCommon Wrong Answer
Relying solely on accuracy as the evaluation metric.
Step 5
Deployment Considerations
How will you ensure the model meets the latency SLA in production?
๐Ÿ’ก Hint
Think about model complexity and inference time.
Key Decision
Choosing a model architecture that balances performance with inference speed.
Interviewer Expectation
Mid: Considers model complexity. Senior: Quantifies expected latency and discusses trade-offs between model size and inference speed.
โš ๏ธCommon Wrong Answer
Focusing only on model accuracy without considering inference speed.
๐Ÿ“Š
Scoring Rubric
โ–ผ Show
Must Have - 60%
  • Identifies overfitting as the primary issue.
  • Proposes a method to reduce model complexity or apply regularization.
  • Addresses class imbalance with appropriate techniques.
  • Chooses precision-recall metrics for evaluation due to data imbalance.
Good to Have - 30%
  • Considers the impact of dataset imbalance on model performance.
  • Evaluates trade-offs between model complexity and latency.
  • Discusses the implications of different regularization techniques.
  • Explains why precision-recall is preferred over accuracy.
Bonus - 10%
  • Quantifies expected latency improvements with proposed changes.
  • Considers the impact of feature engineering constraints on model choice.
  • Proposes a plan for continuous monitoring and adjustment post-deployment.
โœ…
Model Answer
โ–ผ Reveal

The current model is overfitting, as indicated by the high training accuracy and low validation accuracy. To address this, regularization techniques such as pruning or L2 regularization should be applied. Additionally, the class imbalance should be handled using resampling or class weighting. Precision-recall metrics are more suitable for evaluation due to the imbalanced nature of the data. Finally, the model's complexity should be balanced with inference speed to meet the latency SLA.

Key Decisions
  • Identified overfitting due to high training and low validation accuracy.
  • Proposed regularization techniques to reduce model complexity.
  • Suggested handling class imbalance with resampling or class weighting.
  • Chose precision-recall metrics for evaluation due to data imbalance.
  • Considered model complexity and inference speed for latency requirements.
Anti-patterns
Assuming underfitting without considering training accuracy.
Increasing model complexity to improve validation accuracy.
Ignoring class imbalance in the dataset.
Relying solely on accuracy as the evaluation metric.

Practice

(1/5)
1. 
import numpy as np

# Simulate A/B test results
A_results = np.array([1, 0, 1, 1, 0])
B_results = np.array([0, 1, 0, 0, 1])

# Calculate mean conversion rates
A_mean = np.mean(A_results)
B_mean = np.mean(B_results)

# Determine which variant is better
better_variant = 'A' if A_mean > B_mean else 'B'
print(better_variant)
What does this code output?
easy
A. A
B. B
C. Error due to division by zero
D. Error due to shape mismatch

Solution

  1. Step 1: Calculate means

    A_mean = 0.6, B_mean = 0.2

  2. Step 2: Compare means

    A_mean > B_mean, so 'A' is printed.

  3. Final Answer:

    A -> A

  4. Quick Check:

    Calculate means, compare, print. [OK]
Hint: Calculate means, compare, print. [OK]
Common Mistakes:
2. Given the following PyTorch code snippet, what will be the output shape of the gradient for W after backpropagation?
import torch
W = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
x = torch.tensor([1.0, 2.0])
y = W @ x
loss = y.sum()
loss.backward()
easy
A. (2, 2)
B. (2,)
C. (2, 1)
D. (1, 2)

Solution

  1. Step 1: Determine the gradient shape

    The gradient of W will have the same shape as W, which is (2, 2).

  2. Step 2: Eliminate wrong options

    (2,) is incorrect because it suggests a vector shape. (2, 1) and D are incorrect because they suggest mismatched dimensions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Gradient shape matches tensor shape. [OK]
Hint: Gradient shape matches tensor shape. [OK]
Common Mistakes:
3. Given the following Python code snippet, what is the output? 
import numpy as np
relevance_scores = [3, 2, 3, 0, 1, 2]
ideal_dcg = np.sum([3/np.log2(i+2) for i in range(len(relevance_scores))])
print(round(ideal_dcg, 2))
easy
A. 7.86
B. 6.86
C. 8.86
D. 9.86

Solution

  1. Step 1: Calculate the ideal DCG

    Use the formula DCG = ฮฃ(rel_i / log2(i+2)) for each relevance score.

  2. Step 2: Perform the calculation

    3/1 + 2/1.585 + 3/2 + 0/2.322 + 1/2.585 + 2/2.807 = 6.86.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    DCG uses log2(i+2) for position i. [OK]
Hint: DCG uses log2(i+2) for position i. [OK]
Common Mistakes:
4. Given the following Python code, what will be the output when calculating the ROC-AUC for a highly imbalanced dataset?
from sklearn.metrics import roc_auc_score
y_true = [0, 0, 0, 0, 1]
y_scores = [0.1, 0.2, 0.3, 0.4, 0.9]
roc_auc = roc_auc_score(y_true, y_scores)
print(roc_auc)
easy
A. 1.0
B. 0.75
C. 0.9
D. 0.5

Solution

  1. Step 1: Calculate ROC-AUC

    ROC-AUC measures the ability to distinguish between classes. Here, the positive class is perfectly ranked higher than the negative class.

  2. Step 2: Eliminate wrong options

    0.5 would indicate random guessing. 0.75 and 0.9 are incorrect as the positive class is perfectly ranked.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Perfect ranking gives ROC-AUC of 1.0. [OK]
Hint: Perfect ranking gives ROC-AUC of 1.0. [OK]
Common Mistakes:
5. An SVM model with a Gaussian kernel is not performing well on the test set. What is a likely cause?
medium
A. The model is overfitting due to a high regularization parameter
B. The model is underfitting due to a high regularization parameter
C. The model is underfitting due to a low regularization parameter
D. The model is overfitting due to a low regularization parameter

Solution

  1. Step 1: Analyze the effect of regularization on SVM

    A low regularization parameter allows more flexibility, leading to overfitting.

  2. Step 2: Eliminate wrong options

    The model is overfitting due to a high regularization parameter is incorrect because a high regularization parameter reduces overfitting. The model is underfitting due to a low regularization parameter is incorrect because a low regularization parameter does not cause underfitting. The model is underfitting due to a high regularization parameter is incorrect because a high regularization parameter causes underfitting, not overfitting.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Low regularization can lead to overfitting [OK]
Hint: Low regularization can lead to overfitting [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard