Overfitting and underfitting in ML Python - ML Experiment: Train & Evaluate

Experiment - Overfitting and underfitting

Problem:You are training a simple neural network to classify handwritten digits from the MNIST dataset. The current model achieves 98% accuracy on training data but only 75% on validation data.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.85

Issue:The model is overfitting: it performs very well on training data but poorly on unseen validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or preprocessing steps.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

import tensorflow as tf
from tensorflow.keras import layers, models

# Load MNIST dataset
(X_train, y_train), (X_val, y_val) = tf.keras.datasets.mnist.load_data()

# Normalize pixel values
X_train, X_val = X_train / 255.0, X_val / 255.0

# Flatten images
X_train = X_train.reshape(-1, 28*28)
X_val = X_val.reshape(-1, 28*28)

# Build a simpler model with dropout
model = models.Sequential([
    layers.Dense(64, activation='relu', input_shape=(28*28,)),
    layers.Dropout(0.5),
    layers.Dense(32, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Use early stopping
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=30, batch_size=64, validation_data=(X_val, y_val), callbacks=[early_stop])

Reduced number of neurons from larger layers to 64 and 32.

Added dropout layers with 50% dropout rate to reduce overfitting.

Added early stopping to stop training when validation loss stops improving.

Kept learning rate moderate at 0.001 for stable training.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 75%, Training loss 0.05, Validation loss 0.85

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.25, Validation loss 0.35

Adding dropout and early stopping reduces overfitting by preventing the model from memorizing training data. This improves validation accuracy and generalization to new data.

Bonus Experiment

Try reducing the learning rate further and increasing dropout rate to 0.6. Observe how it affects training and validation accuracy.

💡 Hint

Lower learning rate can make training slower but more stable. Higher dropout increases regularization but may reduce training accuracy.

Practice

1. What does overfitting mean in machine learning?

easy

A. The model is too simple and misses important patterns

B. The model learns noise and details too well, failing on new data

C. The model perfectly predicts new data without errors

D. The model ignores training data completely

Solution

1. Step 1: Understand overfitting definition

   Overfitting happens when a model learns not only the true patterns but also the noise and random details in training data.

2. Step 2: Match definition to options

   The model learns noise and details too well, failing on new data describes this exactly, while others describe underfitting or unrelated concepts.

3. Final Answer:

   The model learns noise and details too well, failing on new data -> Option B

4. Quick Check:

   Overfitting = learning noise too well [OK]

Hint: Overfitting means memorizing training noise, not generalizing [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Thinking overfitting means perfect prediction on new data
• Believing overfitting ignores training data

2. Which of the following is a sign of underfitting in a model?

easy

A. High training error but low test error

B. Low training error but high test error

C. Low training error and low test error

D. High training error and high test error

Solution

1. Step 1: Understand underfitting error patterns

   Underfitting means the model is too simple, so it performs poorly on both training and test data, causing high errors.

2. Step 2: Match error patterns to options

   High training error and high test error shows high errors on both training and test sets, which matches underfitting.

3. Final Answer:

   High training error and high test error -> Option D

4. Quick Check:

   Underfitting = high error everywhere [OK]

Hint: Underfitting means bad on training and test data [OK]

Common Mistakes:

• Confusing underfitting with overfitting error patterns
• Assuming low training error means no underfitting
• Ignoring test error when diagnosing underfitting

3. Consider this Python code training a model:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
print(round(train_score, 2), round(test_score, 2))

What is the likely output?

medium

A. 0.60 0.58

B. 0.95 0.90

C. 1.00 0.50

D. 0.80 0.80

Solution

1. Step 1: Understand model complexity and max_depth=1

   A decision tree with max_depth=1 is very simple, likely underfitting the iris data, so training accuracy will be low.

2. Step 2: Predict scores based on underfitting

   Both training and test scores will be low and close, around 0.6, matching 0.60 0.58.

3. Final Answer:

   0.60 0.58 -> Option A

4. Quick Check:

   Simple model = low train and test scores [OK]

Hint: Low max_depth means underfitting, expect low accuracy [OK]

Common Mistakes:

• Assuming max_depth=1 gives perfect accuracy
• Confusing training and test scores
• Ignoring model simplicity effect

4. This code tries to reduce overfitting but has a mistake:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

model = DecisionTreeClassifier(max_depth=10)
model.fit(X_train, y_train)
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

# To reduce overfitting, we set max_depth=1
model.max_depth = 1
print(model.score(X_train, y_train))
print(model.score(X_test, y_test))

What is the error in this approach?

medium

A. Changing max_depth after fitting does not affect the model

B. max_depth=1 causes syntax error

C. fit() method cannot be called twice

D. max_depth must be set to None to reduce overfitting

Solution

1. Step 1: Understand how max_depth affects model

   max_depth is a parameter set when creating the model. Changing it after fitting does not change the trained model.

2. Step 2: Identify mistake in code

   Setting model.max_depth=1 after fitting does not reduce overfitting. The model must be recreated with max_depth=1 before fitting.

3. Final Answer:

   Changing max_depth after fitting does not affect the model -> Option A

4. Quick Check:

   Parameters must be set before fit() [OK]

Hint: Set parameters before fit(), changing after does nothing [OK]

Common Mistakes:

• Thinking changing max_depth after fit changes model
• Believing fit() can only be called once
• Confusing parameter setting with method calls

5. You have a model that overfits training data. Which combination of techniques can help reduce overfitting effectively?

hard

A. Increase model complexity and reduce training data

B. Remove validation set and train longer

C. Use regularization and increase training data size

D. Use fewer features and increase learning rate

Solution

1. Step 1: Identify techniques to reduce overfitting

   Overfitting can be reduced by regularization (penalties to simplify model) and by increasing training data to help generalize better.

2. Step 2: Evaluate options

   Use regularization and increase training data size correctly combines regularization and more data. Other options either increase complexity or remove validation, which worsen overfitting.

3. Final Answer:

   Use regularization and increase training data size -> Option C

4. Quick Check:

   Regularization + more data = less overfitting [OK]

Hint: Regularize and add data to fight overfitting [OK]

Common Mistakes:

• Increasing complexity to fix overfitting
• Ignoring validation data importance
• Increasing learning rate without reason