Overfitting and underfitting in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Overfitting and underfitting

Which metric matters for Overfitting and Underfitting and WHY

To detect overfitting and underfitting, we look at training loss and validation loss or accuracy.

Underfitting happens when both training and validation errors are high. The model is too simple to learn the data well.

Overfitting happens when training error is very low but validation error is high. The model learns training data too well, including noise, and fails on new data.

So, the key metrics are training vs validation loss or accuracy. Their difference tells us if the model is overfitting or underfitting.

Confusion Matrix or Equivalent Visualization

Overfitting and underfitting are best seen in loss or accuracy curves over training steps:

Epoch | Train Loss | Val Loss | Train Acc | Val Acc
-----------------------------------------------
  1   |   0.8     |   0.9   |   60%    |   58%
  5   |   0.3     |   0.7   |   85%    |   65%
 10   |   0.1     |   0.6   |   95%    |   68%
 20   |   0.01    |   0.9   |   99%    |   60%  <-- Overfitting: val loss rises

Underfitting example:
Epoch | Train Loss | Val Loss | Train Acc | Val Acc
-----------------------------------------------
  1   |   0.9     |   0.9   |   55%    |   54%
 10   |   0.8     |   0.85  |   60%    |   58%  <-- Both losses high, accuracy low
    

Precision vs Recall Tradeoff (Analogy for Overfitting vs Underfitting)

Think of overfitting like memorizing answers for one test but failing a new test. The model is too focused on training data details.

Underfitting is like not studying enough for any test, so you do poorly everywhere.

In metrics terms, overfitting means very high training accuracy but low validation accuracy.

Underfitting means low accuracy on both training and validation.

We want a balance: good accuracy on training and validation, meaning the model learned well and generalizes.

What "Good" vs "Bad" Metric Values Look Like

Good:

• Training loss and validation loss both decrease and stay close.
• Training accuracy and validation accuracy both increase and stay close.
• Example: Train Acc 90%, Val Acc 88%

Bad (Overfitting):

• Training loss very low, validation loss starts increasing.
• Training accuracy very high, validation accuracy drops.
• Example: Train Acc 99%, Val Acc 70%

Bad (Underfitting):

• Both training and validation loss high.
• Both training and validation accuracy low.
• Example: Train Acc 60%, Val Acc 58%

Common Pitfalls in Metrics for Overfitting and Underfitting

• Ignoring validation metrics: Only looking at training accuracy can hide overfitting.
• Data leakage: If validation data leaks into training, metrics look falsely good.
• Overfitting to validation: Tuning too much on validation data can cause overfitting to it.
• Using accuracy alone: For imbalanced data, accuracy can be misleading.
• Not using early stopping: Training too long can cause overfitting.

Self-Check Question

Your model has 98% training accuracy but only 12% recall on fraud cases in validation. Is it good for production? Why or why not?

Answer: No, it is not good. The model is overfitting the training data (very high training accuracy) but fails to detect fraud well (very low recall). Missing fraud cases is dangerous, so the model needs improvement to generalize better.

Key Result

Overfitting shows low training loss but high validation loss; underfitting shows high loss on both, so comparing training and validation metrics reveals model fit quality.

Practice

1. What does overfitting mean in machine learning?

easy

A. The model is too simple and misses important patterns

B. The model learns noise and details too well, failing on new data

C. The model perfectly predicts new data without errors

D. The model ignores training data completely

Solution

1. Step 1: Understand overfitting definition

   Overfitting happens when a model learns not only the true patterns but also the noise and random details in training data.

2. Step 2: Match definition to options

   The model learns noise and details too well, failing on new data describes this exactly, while others describe underfitting or unrelated concepts.

3. Final Answer:

   The model learns noise and details too well, failing on new data -> Option B

4. Quick Check:

   Overfitting = learning noise too well [OK]

Hint: Overfitting means memorizing training noise, not generalizing [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Thinking overfitting means perfect prediction on new data
• Believing overfitting ignores training data

2. Which of the following is a sign of underfitting in a model?

easy

A. High training error but low test error

B. Low training error but high test error

C. Low training error and low test error

D. High training error and high test error

Solution

1. Step 1: Understand underfitting error patterns

   Underfitting means the model is too simple, so it performs poorly on both training and test data, causing high errors.

2. Step 2: Match error patterns to options

   High training error and high test error shows high errors on both training and test sets, which matches underfitting.

3. Final Answer:

   High training error and high test error -> Option D

4. Quick Check:

   Underfitting = high error everywhere [OK]

Hint: Underfitting means bad on training and test data [OK]

Common Mistakes:

• Confusing underfitting with overfitting error patterns
• Assuming low training error means no underfitting
• Ignoring test error when diagnosing underfitting

3. Consider this Python code training a model:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(round(train_score, 2), round(test_score, 2))

What is the likely output?

medium

A. 0.60 0.58

B. 0.95 0.90

C. 1.00 0.50

D. 0.80 0.80

Solution

1. Step 1: Understand model complexity and max_depth=1

   A decision tree with max_depth=1 is very simple, likely underfitting the iris data, so training accuracy will be low.

2. Step 2: Predict scores based on underfitting

   Both training and test scores will be low and close, around 0.6, matching 0.60 0.58.

3. Final Answer:

   0.60 0.58 -> Option A

4. Quick Check:

   Simple model = low train and test scores [OK]

Hint: Low max_depth means underfitting, expect low accuracy [OK]

Common Mistakes:

• Assuming max_depth=1 gives perfect accuracy
• Confusing training and test scores
• Ignoring model simplicity effect

4. This code tries to reduce overfitting but has a mistake:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=10)
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

# To reduce overfitting, we set max_depth=1
model.max_depth = 1
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

What is the error in this approach?

medium

A. Changing max_depth after fitting does not affect the model

B. max_depth=1 causes syntax error

C. fit() method cannot be called twice

D. max_depth must be set to None to reduce overfitting

Solution

1. Step 1: Understand how max_depth affects model

   max_depth is a parameter set when creating the model. Changing it after fitting does not change the trained model.

2. Step 2: Identify mistake in code

   Setting model.max_depth=1 after fitting does not reduce overfitting. The model must be recreated with max_depth=1 before fitting.

3. Final Answer:

   Changing max_depth after fitting does not affect the model -> Option A

4. Quick Check:

   Parameters must be set before fit() [OK]

Hint: Set parameters before fit(), changing after does nothing [OK]

Common Mistakes:

• Thinking changing max_depth after fit changes model
• Believing fit() can only be called once
• Confusing parameter setting with method calls

5. You have a model that overfits training data. Which combination of techniques can help reduce overfitting effectively?

hard

A. Increase model complexity and reduce training data

B. Remove validation set and train longer

C. Use regularization and increase training data size

D. Use fewer features and increase learning rate

Solution

1. Step 1: Identify techniques to reduce overfitting

   Overfitting can be reduced by regularization (penalties to simplify model) and by increasing training data to help generalize better.

2. Step 2: Evaluate options

   Use regularization and increase training data size correctly combines regularization and more data. Other options either increase complexity or remove validation, which worsen overfitting.

3. Final Answer:

   Use regularization and increase training data size -> Option C

4. Quick Check:

   Regularization + more data = less overfitting [OK]

Hint: Regularize and add data to fight overfitting [OK]

Common Mistakes:

• Increasing complexity to fix overfitting
• Ignoring validation data importance
• Increasing learning rate without reason