Overfitting and underfitting in ML Python - Deep Dive

Overview - Overfitting and underfitting

What is it?

Overfitting and underfitting describe how well a machine learning model learns from data. Overfitting happens when a model learns too much detail, including noise, making it bad at new data. Underfitting happens when a model learns too little, missing important patterns. Both cause poor predictions on data the model hasn't seen before.

Why it matters

These problems matter because they affect how useful a model is in real life. If a model overfits, it looks perfect on training data but fails in practice. If it underfits, it never learns enough to be helpful. Without understanding these, models would be unreliable, wasting time and resources and possibly causing wrong decisions.

Where it fits

Before learning this, you should know basic machine learning concepts like training data, models, and predictions. After this, you can learn about techniques to fix these problems, like regularization, cross-validation, and model selection.

Mental Model

Core Idea

A good model balances learning enough from data to predict well without memorizing noise or missing key patterns.

Think of it like...

It's like studying for a test: overfitting is memorizing every example question without understanding, so you fail new questions; underfitting is not studying enough, so you don't know the material well.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Underfitting │──────▶│   Good Fit    │──────▶│  Overfitting  │
│ (too simple)  │       │ (balanced)    │       │ (too complex) │
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                      ▲                      ▲
       │                      │                      │
   High error             Low error             High error
   on train & test       on train & test      on train, high on test

Build-Up - 7 Steps

1

Foundation

What is model fitting in ML

Concept: Understanding what it means for a model to fit data.

In machine learning, fitting means the model learns patterns from data. The model looks at input data and tries to find rules to predict outputs. The better it fits, the closer its predictions match the real answers on the training data.

Result

You get a model that can predict outputs for inputs it has seen.

Understanding fitting is the base for knowing why too much or too little fitting causes problems.

2

Foundation

Difference between training and test data

Concept: Introducing the idea of testing model performance on new data.

Training data is what the model learns from. Test data is new data the model hasn't seen. We check how well the model predicts test data to see if it learned general rules or just memorized training examples.

Result

You can measure if a model will work well in real situations.

Knowing the difference helps spot when a model is overfitting or underfitting.

3

Intermediate

What causes underfitting in models

🤔Before reading on: do you think underfitting happens when a model is too simple or too complex? Commit to your answer.

Concept: Underfitting happens when a model is too simple to capture data patterns.

If a model has too few parameters or uses a simple method, it can't learn the true relationships in data. For example, fitting a straight line to data that curves will miss important trends.

Result

The model has high errors on both training and test data.

Understanding underfitting shows why model complexity must match data complexity.

4

Intermediate

What causes overfitting in models

🤔Before reading on: do you think overfitting means the model performs better or worse on training data compared to test data? Commit to your answer.

Concept: Overfitting happens when a model learns noise and details that don't generalize.

A very complex model with many parameters can memorize training data, including random noise. This makes it look perfect on training data but fail on new data because noise is not repeated.

Result

The model has low error on training data but high error on test data.

Knowing overfitting explains why more complexity is not always better.

5

Intermediate

Measuring fit with error metrics

🤔Before reading on: do you think lower training error always means better model? Commit to your answer.

Concept: Using error numbers to see how well a model fits training and test data.

Common metrics like mean squared error or accuracy show how close predictions are to true values. Comparing training and test errors helps detect underfitting (both high) or overfitting (training low, test high).

Result

You can quantify and compare model performance clearly.

Understanding metrics helps diagnose fitting problems objectively.

6

Advanced

Balancing bias and variance tradeoff

🤔Before reading on: do you think bias means error from wrong assumptions or from noise? Commit to your answer.

Concept: Bias and variance explain why models underfit or overfit.

Bias is error from wrong assumptions or too simple models (underfitting). Variance is error from sensitivity to training data noise (overfitting). Good models balance bias and variance to minimize total error.

Result

You understand the fundamental tradeoff behind fitting problems.

Knowing bias-variance tradeoff guides choosing model complexity and training methods.

7

Expert

Why overfitting happens in high dimensions

🤔Before reading on: do you think more features always help model generalize better? Commit to your answer.

Concept: High-dimensional data increases risk of overfitting due to many ways to fit noise.

When data has many features, models can find complex patterns that fit training data perfectly but don't hold for new data. This is called the 'curse of dimensionality'. Regularization and feature selection help control this.

Result

You see why more data features can hurt model generalization without care.

Understanding high-dimensional overfitting explains why data preprocessing is crucial in real projects.

Under the Hood

Models learn by adjusting parameters to reduce error on training data. Overfitting occurs when parameters fit noise, causing complex decision boundaries or functions. Underfitting happens when parameters are too limited to capture true data patterns. The balance depends on model capacity, data size, and noise level.

Why designed this way?

Machine learning models were designed to find patterns in data, but early methods lacked ways to control complexity, causing overfitting. Techniques like regularization and validation sets were introduced to balance learning and generalization. This design reflects the need to handle real-world noisy data effectively.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Data Input  │──────▶│ Model Training│──────▶│ Parameter Fit │
└───────────────┘       └───────────────┘       └───────────────┘
                                │                      │
                                ▼                      ▼
                       ┌───────────────┐      ┌────────────────┐
                       │  Model Output │◀─────│  Error Measure │
                       └───────────────┘      └────────────────┘
                                ▲                      │
                                └──────────────┬───────┘
                                               ▼
                                      Adjust Parameters
                                      (fit better or worse)

Myth Busters - 4 Common Misconceptions

Quick: Does a model with zero training error always perform best on new data? Commit yes or no.

Common Belief: If a model fits training data perfectly, it must be the best model.

Reality: Perfect training fit usually means overfitting, causing poor performance on new data.

Why it matters: Believing this leads to trusting models that fail in real-world use, wasting resources.

Quick: Is underfitting only a problem for very simple models? Commit yes or no.

Common Belief: Underfitting only happens if the model is too simple, like linear regression on complex data.

Reality: Underfitting can also happen if training is stopped too early or features are missing, even in complex models.

Why it matters: Ignoring this can cause missed opportunities to improve models by better training or data.

Quick: Does adding more data always fix overfitting? Commit yes or no.

Common Belief: More data always solves overfitting problems.

Reality: More data helps but doesn't guarantee fixing overfitting if model complexity is too high or noise is large.

Why it matters: Relying only on more data can waste effort without addressing model design issues.

Quick: Can a model be both overfitting and underfitting at the same time? Commit yes or no.

Common Belief: A model cannot be overfitting and underfitting simultaneously.

Reality: A model can underfit some parts of data and overfit others, especially in complex or imbalanced datasets.

Why it matters: Recognizing this helps design better models that handle different data regions properly.

Expert Zone

1

Regularization strength must be carefully tuned; too strong causes underfitting, too weak allows overfitting.

2

Early stopping during training is a practical way to prevent overfitting by monitoring validation error.

3

Data augmentation can reduce overfitting by increasing effective training data diversity without new samples.

When NOT to use

Avoid relying solely on complex models to fix underfitting; sometimes better features or simpler models with proper tuning work best. For overfitting, alternatives include simpler models, pruning, or Bayesian methods instead of just regularization.

Production Patterns

In real systems, cross-validation is used to detect overfitting early. Pipelines include feature selection, regularization, and monitoring test error continuously. Ensembles combine models to reduce overfitting risk.

Connections

Bias-Variance Tradeoff

Overfitting and underfitting are practical outcomes of the bias-variance tradeoff.

Understanding bias and variance helps explain why models fail to generalize and guides model complexity choices.

Human Learning and Memory

Similar to how humans learn, overfitting is like rote memorization, underfitting like not paying attention.

This connection shows that learning well means balancing memorization and understanding, a universal principle.

Signal Processing Noise Filtering

Overfitting is like mistaking noise for signal; underfitting is like filtering out important signals.

Knowing noise filtering helps understand how models must separate true patterns from random fluctuations.

Common Pitfalls

#1 Ignoring test data performance and trusting only training accuracy.

Wrong approach:

model.fit(X_train, y_train)
print('Training accuracy:', model.score(X_train, y_train))
# No test evaluation

Correct approach:

model.fit(X_train, y_train)
print('Training accuracy:', model.score(X_train, y_train))
print('Test accuracy:', model.score(X_test, y_test))

Root cause:

Misunderstanding that training accuracy alone does not reflect real-world performance.

#2 Using an overly complex model on small data without regularization.

Wrong approach:

model = ComplexModel()
model.fit(small_data, labels) # No regularization or validation

Correct approach:

model = ComplexModel(regularization=0.1)
model.fit(small_data, labels)
# Use validation to monitor overfitting

Root cause:

Not recognizing that model complexity must match data size and quality.

#3 Stopping training too early causing underfitting.

Wrong approach:

for epoch in range(1): # Only one epoch
model.train_one_epoch()

Correct approach:

for epoch in range(50): # Train enough epochs
model.train_one_epoch()
if validation_error_increases: break # Early stopping

Root cause:

Misunderstanding training time needed to learn patterns fully.

Key Takeaways

Overfitting means a model learns too much noise, hurting new data predictions; underfitting means it learns too little, missing patterns.

Balancing model complexity and data quality is key to good generalization.

Measuring errors on both training and test data reveals fitting problems clearly.

The bias-variance tradeoff explains why models fail to generalize and guides tuning.

Real-world solutions include regularization, validation, early stopping, and data augmentation.