Overfitting and underfitting in ML Python

Introduction

Overfitting and underfitting explain why a model might not work well on new data. They help us understand if the model learned too much or too little from the training data.

When your model performs very well on training data but poorly on new data.

When your model performs poorly both on training and new data.

When you want to check if your model is too simple or too complex for the problem.

When tuning model settings to improve prediction accuracy.

When deciding how much data or features to use for training.

Syntax

ML Python

No specific code syntax; these are concepts to check during model training and evaluation.

Overfitting means the model memorizes training data details and noise.

Underfitting means the model is too simple to capture the data patterns.

Examples

A very deep decision tree can memorize training data, causing overfitting.

ML Python

# Overfitting example
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=None)  # very deep tree
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)

A very shallow decision tree may be too simple, causing underfitting.

ML Python

# Underfitting example
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=1)  # very shallow tree
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)

Sample Model

This code trains two decision tree models on the iris dataset. One is very deep (likely overfitting), the other is very shallow (likely underfitting). It prints their accuracy on training and test data to show the difference.

ML Python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Overfitting model: very deep tree
model_overfit = DecisionTreeClassifier(max_depth=None, random_state=42)
model_overfit.fit(X_train, y_train)
train_score_overfit = model_overfit.score(X_train, y_train)
test_score_overfit = model_overfit.score(X_test, y_test)

# Underfitting model: very shallow tree
model_underfit = DecisionTreeClassifier(max_depth=1, random_state=42)
model_underfit.fit(X_train, y_train)
train_score_underfit = model_underfit.score(X_train, y_train)
test_score_underfit = model_underfit.score(X_test, y_test)

print(f"Overfitting model - Train accuracy: {train_score_overfit:.2f}")
print(f"Overfitting model - Test accuracy: {test_score_overfit:.2f}")
print(f"Underfitting model - Train accuracy: {train_score_underfit:.2f}")
print(f"Underfitting model - Test accuracy: {test_score_underfit:.2f}")

Important Notes

Overfitting usually shows very high training accuracy but lower test accuracy.

Underfitting shows low accuracy on both training and test data.

Use techniques like cross-validation, pruning, or simpler models to avoid overfitting.

Summary

Overfitting means the model learns too much noise and details from training data.

Underfitting means the model is too simple and misses important patterns.

Good models balance learning enough to predict well on new data without memorizing training data.

Practice

1. What does overfitting mean in machine learning?

easy

A. The model is too simple and misses important patterns

B. The model learns noise and details too well, failing on new data

C. The model perfectly predicts new data without errors

D. The model ignores training data completely

Solution

1. Step 1: Understand overfitting definition

   Overfitting happens when a model learns not only the true patterns but also the noise and random details in training data.

2. Step 2: Match definition to options

   The model learns noise and details too well, failing on new data describes this exactly, while others describe underfitting or unrelated concepts.

3. Final Answer:

   The model learns noise and details too well, failing on new data -> Option B

4. Quick Check:

   Overfitting = learning noise too well [OK]

Hint: Overfitting means memorizing training noise, not generalizing [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Thinking overfitting means perfect prediction on new data
• Believing overfitting ignores training data

2. Which of the following is a sign of underfitting in a model?

easy

A. High training error but low test error

B. Low training error but high test error

C. Low training error and low test error

D. High training error and high test error

Solution

1. Step 1: Understand underfitting error patterns

   Underfitting means the model is too simple, so it performs poorly on both training and test data, causing high errors.

2. Step 2: Match error patterns to options

   High training error and high test error shows high errors on both training and test sets, which matches underfitting.

3. Final Answer:

   High training error and high test error -> Option D

4. Quick Check:

   Underfitting = high error everywhere [OK]

Hint: Underfitting means bad on training and test data [OK]

Common Mistakes:

• Confusing underfitting with overfitting error patterns
• Assuming low training error means no underfitting
• Ignoring test error when diagnosing underfitting

3. Consider this Python code training a model:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(round(train_score, 2), round(test_score, 2))

What is the likely output?

medium

A. 0.60 0.58

B. 0.95 0.90

C. 1.00 0.50

D. 0.80 0.80

Solution

1. Step 1: Understand model complexity and max_depth=1

   A decision tree with max_depth=1 is very simple, likely underfitting the iris data, so training accuracy will be low.

2. Step 2: Predict scores based on underfitting

   Both training and test scores will be low and close, around 0.6, matching 0.60 0.58.

3. Final Answer:

   0.60 0.58 -> Option A

4. Quick Check:

   Simple model = low train and test scores [OK]

Hint: Low max_depth means underfitting, expect low accuracy [OK]

Common Mistakes:

• Assuming max_depth=1 gives perfect accuracy
• Confusing training and test scores
• Ignoring model simplicity effect

4. This code tries to reduce overfitting but has a mistake:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=10)
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

# To reduce overfitting, we set max_depth=1
model.max_depth = 1
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

What is the error in this approach?

medium

A. Changing max_depth after fitting does not affect the model

B. max_depth=1 causes syntax error

C. fit() method cannot be called twice

D. max_depth must be set to None to reduce overfitting

Solution

1. Step 1: Understand how max_depth affects model

   max_depth is a parameter set when creating the model. Changing it after fitting does not change the trained model.

2. Step 2: Identify mistake in code

   Setting model.max_depth=1 after fitting does not reduce overfitting. The model must be recreated with max_depth=1 before fitting.

3. Final Answer:

   Changing max_depth after fitting does not affect the model -> Option A

4. Quick Check:

   Parameters must be set before fit() [OK]

Hint: Set parameters before fit(), changing after does nothing [OK]

Common Mistakes:

• Thinking changing max_depth after fit changes model
• Believing fit() can only be called once
• Confusing parameter setting with method calls

5. You have a model that overfits training data. Which combination of techniques can help reduce overfitting effectively?

hard

A. Increase model complexity and reduce training data

B. Remove validation set and train longer

C. Use regularization and increase training data size

D. Use fewer features and increase learning rate

Solution

1. Step 1: Identify techniques to reduce overfitting

   Overfitting can be reduced by regularization (penalties to simplify model) and by increasing training data to help generalize better.

2. Step 2: Evaluate options

   Use regularization and increase training data size correctly combines regularization and more data. Other options either increase complexity or remove validation, which worsen overfitting.

3. Final Answer:

   Use regularization and increase training data size -> Option C

4. Quick Check:

   Regularization + more data = less overfitting [OK]

Hint: Regularize and add data to fight overfitting [OK]

Common Mistakes:

• Increasing complexity to fix overfitting
• Ignoring validation data importance
• Increasing learning rate without reason