Experiment - First neural network

Problem:Build a simple neural network to classify handwritten digits from the MNIST dataset.

Current Metrics:Training accuracy: 98%, Validation accuracy: 85%

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 90% while keeping training accuracy below 95%.

You can only change the model architecture and training parameters.

Do not change the dataset or preprocessing steps.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load MNIST dataset
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

# Normalize pixel values
X_train, X_test = X_train / 255.0, X_test / 255.0

# Build model with dropout and smaller hidden layer
model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

# Compile model with lower learning rate
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# Use early stopping callback
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# Train model
history = model.fit(
    X_train, y_train,
    epochs=30,
    batch_size=64,
    validation_split=0.2,
    callbacks=[early_stop],
    verbose=0
)

# Evaluate on test data
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)

print(f'Test accuracy: {test_acc * 100:.2f}%')

Added a Dropout layer with rate 0.3 after the first Dense layer to reduce overfitting.

Reduced the number of neurons in the hidden Dense layer from 128 to 64.

Lowered the learning rate to 0.001 for more stable training.

Added early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 85%

After: Training accuracy: 93%, Validation accuracy: 91%

Adding dropout, reducing model size, lowering learning rate, and using early stopping help reduce overfitting and improve validation accuracy.

Bonus Experiment

Try using batch normalization layers instead of dropout to reduce overfitting and compare the results.

💡 Hint

Insert batch normalization layers after Dense layers and observe if validation accuracy improves.