Overfitting and underfitting in ML Python - Deep Dive

Overview - Overfitting and underfitting

What is it?

Overfitting and underfitting describe how well a machine learning model learns from data. Overfitting happens when a model learns too much detail, including noise, making it bad at new data. Underfitting happens when a model learns too little, missing important patterns. Both cause poor predictions on data the model hasn't seen before.

Why it matters

These problems matter because they affect how useful a model is in real life. If a model overfits, it looks perfect on training data but fails in practice. If it underfits, it never learns enough to be helpful. Without understanding these, models would be unreliable, wasting time and resources and possibly causing wrong decisions.

Where it fits

Before learning this, you should know basic machine learning concepts like training data, models, and predictions. After this, you can learn about techniques to fix these problems, like regularization, cross-validation, and model selection.

Mental Model

Core Idea

A good model balances learning enough from data to predict well without memorizing noise or missing key patterns.

Think of it like...

It's like studying for a test: overfitting is memorizing every example question without understanding, so you fail new questions; underfitting is not studying enough, so you don't know the material well.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Underfitting │──────▶│   Good Fit    │──────▶│  Overfitting  │
│ (too simple)  │       │ (balanced)    │       │ (too complex) │
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                      ▲                      ▲
       │                      │                      │
   High error             Low error             High error
   on train & test       on train & test      on train, high on test

Build-Up - 7 Steps

1

Foundation

What is model fitting in ML

Concept: Understanding what it means for a model to fit data.

In machine learning, fitting means the model learns patterns from data. The model looks at input data and tries to find rules to predict outputs. The better it fits, the closer its predictions match the real answers on the training data.

Result

You get a model that can predict outputs for inputs it has seen.

Understanding fitting is the base for knowing why too much or too little fitting causes problems.

2

Foundation

Difference between training and test data

Concept: Introducing the idea of testing model performance on new data.

Training data is what the model learns from. Test data is new data the model hasn't seen. We check how well the model predicts test data to see if it learned general rules or just memorized training examples.

Result

You can measure if a model will work well in real situations.

Knowing the difference helps spot when a model is overfitting or underfitting.

3

Intermediate

What causes underfitting in models

🤔Before reading on: do you think underfitting happens when a model is too simple or too complex? Commit to your answer.

Concept: Underfitting happens when a model is too simple to capture data patterns.

If a model has too few parameters or uses a simple method, it can't learn the true relationships in data. For example, fitting a straight line to data that curves will miss important trends.

Result

The model has high errors on both training and test data.

Understanding underfitting shows why model complexity must match data complexity.

4

Intermediate

What causes overfitting in models

🤔Before reading on: do you think overfitting means the model performs better or worse on training data compared to test data? Commit to your answer.

Concept: Overfitting happens when a model learns noise and details that don't generalize.

A very complex model with many parameters can memorize training data, including random noise. This makes it look perfect on training data but fail on new data because noise is not repeated.

Result

The model has low error on training data but high error on test data.

Knowing overfitting explains why more complexity is not always better.

5

Intermediate

Measuring fit with error metrics

🤔Before reading on: do you think lower training error always means better model? Commit to your answer.

Concept: Using error numbers to see how well a model fits training and test data.

Common metrics like mean squared error or accuracy show how close predictions are to true values. Comparing training and test errors helps detect underfitting (both high) or overfitting (training low, test high).

Result

You can quantify and compare model performance clearly.

Understanding metrics helps diagnose fitting problems objectively.

6

Advanced

Balancing bias and variance tradeoff

🤔Before reading on: do you think bias means error from wrong assumptions or from noise? Commit to your answer.

Concept: Bias and variance explain why models underfit or overfit.

Bias is error from wrong assumptions or too simple models (underfitting). Variance is error from sensitivity to training data noise (overfitting). Good models balance bias and variance to minimize total error.

Result

You understand the fundamental tradeoff behind fitting problems.

Knowing bias-variance tradeoff guides choosing model complexity and training methods.

7

Expert

Why overfitting happens in high dimensions

🤔Before reading on: do you think more features always help model generalize better? Commit to your answer.

Concept: High-dimensional data increases risk of overfitting due to many ways to fit noise.

When data has many features, models can find complex patterns that fit training data perfectly but don't hold for new data. This is called the 'curse of dimensionality'. Regularization and feature selection help control this.

Result

You see why more data features can hurt model generalization without care.

Understanding high-dimensional overfitting explains why data preprocessing is crucial in real projects.

Under the Hood

Models learn by adjusting parameters to reduce error on training data. Overfitting occurs when parameters fit noise, causing complex decision boundaries or functions. Underfitting happens when parameters are too limited to capture true data patterns. The balance depends on model capacity, data size, and noise level.

Why designed this way?

Machine learning models were designed to find patterns in data, but early methods lacked ways to control complexity, causing overfitting. Techniques like regularization and validation sets were introduced to balance learning and generalization. This design reflects the need to handle real-world noisy data effectively.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Data Input  │──────▶│ Model Training│──────▶│ Parameter Fit │
└───────────────┘       └───────────────┘       └───────────────┘
                                │                      │
                                ▼                      ▼
                       ┌───────────────┐      ┌────────────────┐
                       │  Model Output │◀─────│  Error Measure │
                       └───────────────┘      └────────────────┘
                                ▲                      │
                                └──────────────┬───────┘
                                               ▼
                                      Adjust Parameters
                                      (fit better or worse)

Myth Busters - 4 Common Misconceptions

Quick: Does a model with zero training error always perform best on new data? Commit yes or no.

Common Belief: If a model fits training data perfectly, it must be the best model.

Reality: Perfect training fit usually means overfitting, causing poor performance on new data.

Why it matters: Believing this leads to trusting models that fail in real-world use, wasting resources.

Quick: Is underfitting only a problem for very simple models? Commit yes or no.

Common Belief: Underfitting only happens if the model is too simple, like linear regression on complex data.

Reality: Underfitting can also happen if training is stopped too early or features are missing, even in complex models.

Why it matters: Ignoring this can cause missed opportunities to improve models by better training or data.

Quick: Does adding more data always fix overfitting? Commit yes or no.

Common Belief: More data always solves overfitting problems.

Reality: More data helps but doesn't guarantee fixing overfitting if model complexity is too high or noise is large.

Why it matters: Relying only on more data can waste effort without addressing model design issues.

Quick: Can a model be both overfitting and underfitting at the same time? Commit yes or no.

Common Belief: A model cannot be overfitting and underfitting simultaneously.

Reality: A model can underfit some parts of data and overfit others, especially in complex or imbalanced datasets.

Why it matters: Recognizing this helps design better models that handle different data regions properly.

Expert Zone

1

Regularization strength must be carefully tuned; too strong causes underfitting, too weak allows overfitting.

2

Early stopping during training is a practical way to prevent overfitting by monitoring validation error.

3

Data augmentation can reduce overfitting by increasing effective training data diversity without new samples.

When NOT to use

Avoid relying solely on complex models to fix underfitting; sometimes better features or simpler models with proper tuning work best. For overfitting, alternatives include simpler models, pruning, or Bayesian methods instead of just regularization.

Production Patterns

In real systems, cross-validation is used to detect overfitting early. Pipelines include feature selection, regularization, and monitoring test error continuously. Ensembles combine models to reduce overfitting risk.

Connections

Bias-Variance Tradeoff

Overfitting and underfitting are practical outcomes of the bias-variance tradeoff.

Understanding bias and variance helps explain why models fail to generalize and guides model complexity choices.

Human Learning and Memory

Similar to how humans learn, overfitting is like rote memorization, underfitting like not paying attention.

This connection shows that learning well means balancing memorization and understanding, a universal principle.

Signal Processing Noise Filtering

Overfitting is like mistaking noise for signal; underfitting is like filtering out important signals.

Knowing noise filtering helps understand how models must separate true patterns from random fluctuations.

Common Pitfalls

#1 Ignoring test data performance and trusting only training accuracy.

Wrong approach:

model.fit(X_train, y_train)
print('Training accuracy:', model.score(X_train, y_train))
# No test evaluation

Correct approach:

model.fit(X_train, y_train)
print('Training accuracy:', model.score(X_train, y_train))
print('Test accuracy:', model.score(X_test, y_test))

Root cause:

Misunderstanding that training accuracy alone does not reflect real-world performance.

#2 Using an overly complex model on small data without regularization.

Wrong approach:

model = ComplexModel()
model.fit(small_data, labels) # No regularization or validation

Correct approach:

model = ComplexModel(regularization=0.1)
model.fit(small_data, labels)
# Use validation to monitor overfitting

Root cause:

Not recognizing that model complexity must match data size and quality.

#3 Stopping training too early causing underfitting.

Wrong approach:

for epoch in range(1): # Only one epoch
model.train_one_epoch()

Correct approach:

for epoch in range(50): # Train enough epochs
model.train_one_epoch()
if validation_error_increases: break # Early stopping

Root cause:

Misunderstanding training time needed to learn patterns fully.

Key Takeaways

Overfitting means a model learns too much noise, hurting new data predictions; underfitting means it learns too little, missing patterns.

Balancing model complexity and data quality is key to good generalization.

Measuring errors on both training and test data reveals fitting problems clearly.

The bias-variance tradeoff explains why models fail to generalize and guides tuning.

Real-world solutions include regularization, validation, early stopping, and data augmentation.

Practice

1. What does overfitting mean in machine learning?

easy

A. The model is too simple and misses important patterns

B. The model learns noise and details too well, failing on new data

C. The model perfectly predicts new data without errors

D. The model ignores training data completely

Solution

1. Step 1: Understand overfitting definition

   Overfitting happens when a model learns not only the true patterns but also the noise and random details in training data.

2. Step 2: Match definition to options

   The model learns noise and details too well, failing on new data describes this exactly, while others describe underfitting or unrelated concepts.

3. Final Answer:

   The model learns noise and details too well, failing on new data -> Option B

4. Quick Check:

   Overfitting = learning noise too well [OK]

Hint: Overfitting means memorizing training noise, not generalizing [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Thinking overfitting means perfect prediction on new data
• Believing overfitting ignores training data

2. Which of the following is a sign of underfitting in a model?

easy

A. High training error but low test error

B. Low training error but high test error

C. Low training error and low test error

D. High training error and high test error

Solution

1. Step 1: Understand underfitting error patterns

   Underfitting means the model is too simple, so it performs poorly on both training and test data, causing high errors.

2. Step 2: Match error patterns to options

   High training error and high test error shows high errors on both training and test sets, which matches underfitting.

3. Final Answer:

   High training error and high test error -> Option D

4. Quick Check:

   Underfitting = high error everywhere [OK]

Hint: Underfitting means bad on training and test data [OK]

Common Mistakes:

• Confusing underfitting with overfitting error patterns
• Assuming low training error means no underfitting
• Ignoring test error when diagnosing underfitting

3. Consider this Python code training a model:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(round(train_score, 2), round(test_score, 2))

What is the likely output?

medium

A. 0.60 0.58

B. 0.95 0.90

C. 1.00 0.50

D. 0.80 0.80

Solution

1. Step 1: Understand model complexity and max_depth=1

   A decision tree with max_depth=1 is very simple, likely underfitting the iris data, so training accuracy will be low.

2. Step 2: Predict scores based on underfitting

   Both training and test scores will be low and close, around 0.6, matching 0.60 0.58.

3. Final Answer:

   0.60 0.58 -> Option A

4. Quick Check:

   Simple model = low train and test scores [OK]

Hint: Low max_depth means underfitting, expect low accuracy [OK]

Common Mistakes:

• Assuming max_depth=1 gives perfect accuracy
• Confusing training and test scores
• Ignoring model simplicity effect

4. This code tries to reduce overfitting but has a mistake:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=10)
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

# To reduce overfitting, we set max_depth=1
model.max_depth = 1
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

What is the error in this approach?

medium

A. Changing max_depth after fitting does not affect the model

B. max_depth=1 causes syntax error

C. fit() method cannot be called twice

D. max_depth must be set to None to reduce overfitting

Solution

1. Step 1: Understand how max_depth affects model

   max_depth is a parameter set when creating the model. Changing it after fitting does not change the trained model.

2. Step 2: Identify mistake in code

   Setting model.max_depth=1 after fitting does not reduce overfitting. The model must be recreated with max_depth=1 before fitting.

3. Final Answer:

   Changing max_depth after fitting does not affect the model -> Option A

4. Quick Check:

   Parameters must be set before fit() [OK]

Hint: Set parameters before fit(), changing after does nothing [OK]

Common Mistakes:

• Thinking changing max_depth after fit changes model
• Believing fit() can only be called once
• Confusing parameter setting with method calls

5. You have a model that overfits training data. Which combination of techniques can help reduce overfitting effectively?

hard

A. Increase model complexity and reduce training data

B. Remove validation set and train longer

C. Use regularization and increase training data size

D. Use fewer features and increase learning rate

Solution

1. Step 1: Identify techniques to reduce overfitting

   Overfitting can be reduced by regularization (penalties to simplify model) and by increasing training data to help generalize better.

2. Step 2: Evaluate options

   Use regularization and increase training data size correctly combines regularization and more data. Other options either increase complexity or remove validation, which worsen overfitting.

3. Final Answer:

   Use regularization and increase training data size -> Option C

4. Quick Check:

   Regularization + more data = less overfitting [OK]

Hint: Regularize and add data to fight overfitting [OK]

Common Mistakes:

• Increasing complexity to fix overfitting
• Ignoring validation data importance
• Increasing learning rate without reason