Bird
Raised Fist0

Overfitting and Underfitting - Diagnosing and Fixing

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Overfitting vs Underfitting

When tuning a model, how do you decide whether to address overfitting or underfitting, and what steps would you take for each?

📊
Comparison
DimensionOverfittingUnderfitting
Training accuracyHighLow
Validation accuracyLowLow
Model complexityHighLow
GeneralizationLowHigh
Sensitivity to noiseHighLow
✅
Verdict
Default to addressing overfitting when training accuracy is high but validation accuracy is low; switch to addressing underfitting when both training and validation accuracies are low, indicating the model is too simple.
💡
Key Insight
Balancing model complexity and data size is crucial to avoid both overfitting and underfitting.
⚠️
Interview Trap
❌ Wrong: Always reduce model complexity to avoid overfitting.
Ignoring the fact that reducing complexity can lead to underfitting, especially with complex data.
✅ Evaluate both training and validation performance to decide whether to simplify the model or enhance it with more features or complexity.
Overfitting
"Captures complex patterns in the training data, potentially leading to high training accuracy."
Weaknesses
  • Poor generalization to unseen data.
  • High variance in model predictions.
  • Requires more regularization techniques.
  • Sensitive to noise in the training data.
Ideal When
  • Training accuracy > 95% but validation accuracy < 80%
  • Model complexity is high (e.g., deep neural networks with many parameters)
  • Training data size is small (n < 10K)
  • High feature-to-sample ratio
Google's early image recognition models often faced overfitting due to limited labeled data and high model complexity.
Underfitting
"Ensures model simplicity, reducing the risk of capturing noise."
Weaknesses
  • Fails to capture underlying patterns in the data.
  • Low training and validation accuracy.
  • High bias in model predictions.
  • Limited by model capacity.
Ideal When
  • Training accuracy < 70%
  • Model complexity is low (e.g., linear models with few parameters)
  • Large training data size (n > 100K)
  • Low feature-to-sample ratio
Amazon's recommendation systems initially underfit due to overly simplistic models that couldn't capture user preferences.
🐍
Overfitting - Example
▼ Show
Python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

X, y = np.random.rand(100, 10), np.random.rand(100)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = LinearRegression().fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(f'Train: {train_score}, Test: {test_score}')
🐍
Underfitting - Example
▼ Show
Python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

X, y = np.random.rand(1000, 2), np.random.rand(1000)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = LinearRegression().fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(f'Train: {train_score}, Test: {test_score}')
✅
Use Overfitting When
  • •Training accuracy > 90% and validation accuracy < 80%
  • •Model complexity is high with many parameters
  • •Training data size is small (n < 10K)
  • •High feature-to-sample ratio
✅
Use Underfitting When
  • •Training accuracy < 70%
  • •Model complexity is low with few parameters
  • •Large training data size (n > 100K)
  • •Low feature-to-sample ratio
↳
Nuances
  • Overfitting can sometimes be mitigated by data augmentation, which is not applicable for underfitting.
  • Regularization techniques like L1/L2 can help with overfitting but may worsen underfitting.
  • Increasing model complexity can solve underfitting but exacerbate overfitting if not controlled.
  • Cross-validation can help detect overfitting but may not always indicate underfitting.

Practice

(1/5)
1. Given the following Python code, what will be the output? 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(len(X_train), len(X_test))
easy
A. 6 4
B. 8 2
C. 7 3
D. 5 5

Solution

  1. Step 1: Calculate the split sizes

    With a test_size of 0.2, 20% of 10 samples go to the test set, i.e., 2 samples.

  2. Step 2: Determine the training set size

    The remaining 80% (8 samples) go to the training set.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    80% train, 20% test split. [OK]
Hint: 80% train, 20% test split. [OK]
Common Mistakes:
2. When comparing two ML models using A/B testing, why might you choose causal inference over the holdout method?
medium
A. Holdout method is less accurate.
B. Causal inference is faster to implement.
C. Causal inference provides insights into why one model performs better.
D. Holdout method cannot be used with large datasets.

Solution

  1. Step 1: Understand the methods

    Causal inference helps understand the cause-effect relationship, not just performance.

  2. Step 2: Eliminate wrong options

    Holdout is not inherently less accurate or unsuitable for large datasets. Causal inference is not necessarily faster.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Causal inference explains why, not just what. [OK]
Hint: Causal inference explains why, not just what. [OK]
Common Mistakes:
3. When choosing between K-Means and Gaussian Mixture Models (GMM) for clustering, which scenario favors K-Means?
medium
A. Data with overlapping clusters
B. Clusters with different covariance structures
C. Clusters with varying densities
D. Clusters with similar variance and spherical shape

Solution

  1. Step 1: Compare K-Means and GMM

    K-Means assumes spherical clusters with similar variance, while GMM can handle different shapes and densities.

  2. Step 2: Identify the scenario

    Clusters with similar variance and spherical shape describes clusters with similar variance and spherical shape, which is ideal for K-Means.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    K-Means loves spherical, similar variance clusters [OK]
Hint: K-Means loves spherical, similar variance clusters [OK]
Common Mistakes:
4. When deciding between offline and online evaluation for a new feature in a web application, which approach is more suitable if user feedback is critical and why?
medium
A. Offline evaluation, because it allows for controlled experiments.
B. Online evaluation, because it captures real user feedback and interaction.
C. Offline evaluation, because it is faster and less resource-intensive.
D. Online evaluation, because it avoids overfitting to historical data.

Solution

  1. Step 1: Identify the need for user feedback.

    When user feedback is critical, capturing real interactions is essential.

  2. Step 2: Evaluate the suitability of each approach.

    Offline evaluation, because it allows for controlled experiments. is incorrect because controlled experiments do not capture real-time feedback. Offline evaluation, because it is faster and less resource-intensive. is incorrect because speed and resource usage do not address feedback needs. Online evaluation, because it avoids overfitting to historical data. is partially correct but does not emphasize feedback.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Real feedback needs online evaluation. [OK]
Hint: Real feedback needs online evaluation. [OK]
Common Mistakes:
5. A model's performance improved significantly on the test set but not on the validation set. What is the most likely cause?
medium
A. There is data leakage from the test set.
B. The test set is not representative of the validation set.
C. The model is overfitting to the training data.
D. The validation set is too small to detect improvements.

Solution

  1. Step 1: Analyze performance discrepancy

    Significant improvement on the test set but not on the validation set suggests an issue with test set integrity.

  2. Step 2: Identify data leakage

    Data leakage from the test set can artificially inflate performance metrics.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Test set improvement only? Check for leakage [OK]
Hint: Test set improvement only? Check for leakage [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard