Why Overfitting and underfitting in ML Python? - Purpose & Use Cases

The Big Idea

What if your model is just memorizing instead of truly learning? Discover how to fix that!

The Scenario

Imagine you are trying to guess the pattern of a friend's drawing by memorizing every tiny detail instead of understanding the overall shape.

The Problem

When you memorize every detail, you get confused by small changes and can't guess new drawings well. If you guess too simple, you miss important details. Both ways make your guesses wrong often.

The Solution

Understanding overfitting and underfitting helps you find the right balance: learning enough details to be accurate but not so much that you get confused by noise.

Before vs After

✗ Before

model.fit(data)
# memorizes all details, fails on new data

✓ After

model.fit(data, regularization=True)
# learns main patterns, ignores noise

What It Enables

It lets models make smart predictions that work well on new, unseen data, not just the examples they learned from.

Real Life Example

Like a student who understands math concepts well enough to solve new problems, not just repeat memorized answers.

Key Takeaways

Overfitting means learning too much noise, causing poor new predictions.

Underfitting means learning too little, missing important patterns.

Balancing these helps models predict accurately on new data.

Practice

1. What does overfitting mean in machine learning?

easy

A. The model is too simple and misses important patterns

B. The model learns noise and details too well, failing on new data

C. The model perfectly predicts new data without errors

D. The model ignores training data completely

Solution

1. Step 1: Understand overfitting definition

   Overfitting happens when a model learns not only the true patterns but also the noise and random details in training data.

2. Step 2: Match definition to options

   The model learns noise and details too well, failing on new data describes this exactly, while others describe underfitting or unrelated concepts.

3. Final Answer:

   The model learns noise and details too well, failing on new data -> Option B

4. Quick Check:

   Overfitting = learning noise too well [OK]

Hint: Overfitting means memorizing training noise, not generalizing [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Thinking overfitting means perfect prediction on new data
• Believing overfitting ignores training data

2. Which of the following is a sign of underfitting in a model?

easy

A. High training error but low test error

B. Low training error but high test error

C. Low training error and low test error

D. High training error and high test error

Solution

1. Step 1: Understand underfitting error patterns

   Underfitting means the model is too simple, so it performs poorly on both training and test data, causing high errors.

2. Step 2: Match error patterns to options

   High training error and high test error shows high errors on both training and test sets, which matches underfitting.

3. Final Answer:

   High training error and high test error -> Option D

4. Quick Check:

   Underfitting = high error everywhere [OK]

Hint: Underfitting means bad on training and test data [OK]

Common Mistakes:

• Confusing underfitting with overfitting error patterns
• Assuming low training error means no underfitting
• Ignoring test error when diagnosing underfitting

3. Consider this Python code training a model:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(round(train_score, 2), round(test_score, 2))

What is the likely output?

medium

A. 0.60 0.58

B. 0.95 0.90

C. 1.00 0.50

D. 0.80 0.80

Solution

1. Step 1: Understand model complexity and max_depth=1

   A decision tree with max_depth=1 is very simple, likely underfitting the iris data, so training accuracy will be low.

2. Step 2: Predict scores based on underfitting

   Both training and test scores will be low and close, around 0.6, matching 0.60 0.58.

3. Final Answer:

   0.60 0.58 -> Option A

4. Quick Check:

   Simple model = low train and test scores [OK]

Hint: Low max_depth means underfitting, expect low accuracy [OK]

Common Mistakes:

• Assuming max_depth=1 gives perfect accuracy
• Confusing training and test scores
• Ignoring model simplicity effect

4. This code tries to reduce overfitting but has a mistake:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=10)
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

# To reduce overfitting, we set max_depth=1
model.max_depth = 1
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

What is the error in this approach?

medium

A. Changing max_depth after fitting does not affect the model

B. max_depth=1 causes syntax error

C. fit() method cannot be called twice

D. max_depth must be set to None to reduce overfitting

Solution

1. Step 1: Understand how max_depth affects model

   max_depth is a parameter set when creating the model. Changing it after fitting does not change the trained model.

2. Step 2: Identify mistake in code

   Setting model.max_depth=1 after fitting does not reduce overfitting. The model must be recreated with max_depth=1 before fitting.

3. Final Answer:

   Changing max_depth after fitting does not affect the model -> Option A

4. Quick Check:

   Parameters must be set before fit() [OK]

Hint: Set parameters before fit(), changing after does nothing [OK]

Common Mistakes:

• Thinking changing max_depth after fit changes model
• Believing fit() can only be called once
• Confusing parameter setting with method calls

5. You have a model that overfits training data. Which combination of techniques can help reduce overfitting effectively?

hard

A. Increase model complexity and reduce training data

B. Remove validation set and train longer

C. Use regularization and increase training data size

D. Use fewer features and increase learning rate

Solution

1. Step 1: Identify techniques to reduce overfitting

   Overfitting can be reduced by regularization (penalties to simplify model) and by increasing training data to help generalize better.

2. Step 2: Evaluate options

   Use regularization and increase training data size correctly combines regularization and more data. Other options either increase complexity or remove validation, which worsen overfitting.

3. Final Answer:

   Use regularization and increase training data size -> Option C

4. Quick Check:

   Regularization + more data = less overfitting [OK]

Hint: Regularize and add data to fight overfitting [OK]

Common Mistakes:

• Increasing complexity to fix overfitting
• Ignoring validation data importance
• Increasing learning rate without reason