Experiment - Model comparison

Problem:Classify images of handwritten digits using two different neural network models and compare their performance.

Current Metrics:Model A (Simple CNN): Training accuracy 98%, Validation accuracy 90%. Model B (Deeper CNN): Training accuracy 99%, Validation accuracy 88%.

Issue:Model B overfits the training data, showing higher training accuracy but lower validation accuracy compared to Model A.

Your Task

Reduce overfitting in Model B so that its validation accuracy improves to at least 92%, while keeping training accuracy below 97%.

You can only modify Model B architecture or training parameters.

Do not change Model A.

Use the MNIST dataset for training and validation.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical

# Load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalize and reshape
X_train = X_train.reshape(-1, 28, 28, 1).astype('float32') / 255
X_test = X_test.reshape(-1, 28, 28, 1).astype('float32') / 255

y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)

# Define improved Model B with dropout
model_b = models.Sequential([
    layers.Conv2D(32, (3,3), activation='relu', input_shape=(28,28,1)),
    layers.MaxPooling2D((2,2)),
    layers.Conv2D(64, (3,3), activation='relu'),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model_b.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Early stopping callback
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# Train model
history = model_b.fit(X_train, y_train, epochs=30, batch_size=64, validation_split=0.2, callbacks=[early_stop], verbose=0)

# Evaluate
train_loss, train_acc = model_b.evaluate(X_train, y_train, verbose=0)
val_loss, val_acc = model_b.evaluate(X_test, y_test, verbose=0)

print(f'Training accuracy: {train_acc*100:.2f}%')
print(f'Validation accuracy: {val_acc*100:.2f}%')

Added dropout layers after convolutional and dense layers to reduce overfitting.

Implemented early stopping to stop training when validation loss stops improving.

Kept optimizer as Adam and batch size at 64 for stable training.

Results Interpretation

Before: Model B had 99% training accuracy but only 88% validation accuracy, showing overfitting.

After: With dropout and early stopping, training accuracy dropped to 96.5%, but validation accuracy improved to 93.2%, indicating better generalization.

Adding dropout and early stopping helps reduce overfitting by preventing the model from memorizing training data, leading to better performance on new data.

Bonus Experiment

Try replacing dropout with batch normalization layers in Model B and compare the validation accuracy.

💡 Hint

Batch normalization can stabilize and speed up training, sometimes improving generalization without dropout.