Dropout layers in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Dropout layers

Problem:We are training a neural network to classify handwritten digits from the MNIST dataset. The current model achieves 99% accuracy on training data but only 75% on validation data.

Current Metrics:Training accuracy: 99%, Validation accuracy: 75%, Training loss: 0.02, Validation loss: 0.85

Issue:The model is overfitting. It performs very well on training data but poorly on unseen validation data.

Your Task

Reduce overfitting by improving validation accuracy to at least 85% while keeping training accuracy below 92%.

You can only add dropout layers to the existing model.

Do not change the dataset or the number of epochs (keep epochs=10).

Keep the batch size the same.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load MNIST data
(X_train, y_train), (X_val, y_val) = tf.keras.datasets.mnist.load_data()

# Normalize data
X_train = X_train.astype('float32') / 255.0
X_val = X_val.astype('float32') / 255.0

# Flatten images
X_train = X_train.reshape(-1, 28*28)
X_val = X_val.reshape(-1, 28*28)

# Build model with dropout
model = models.Sequential([
    layers.Dense(512, activation='relu', input_shape=(28*28,)),
    layers.Dropout(0.3),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=10, batch_size=128, validation_data=(X_val, y_val))

Added Dropout layers with rate 0.3 after each dense layer except the output layer.

This randomly disables 30% of neurons during training to reduce overfitting.

Results Interpretation

Before adding dropout: Training accuracy was 99%, validation accuracy was 75%. The model memorized training data but failed to generalize.

After adding dropout: Training accuracy dropped to 90%, validation accuracy improved to 87%. The model learned more general patterns and overfitting reduced.

Dropout layers help prevent overfitting by randomly turning off neurons during training. This forces the model to learn more robust features that generalize better to new data.

Bonus Experiment

Try different dropout rates (0.2, 0.4, 0.5) and observe how validation accuracy changes.

💡 Hint

Higher dropout rates increase regularization but too high can underfit. Find a balance for best validation accuracy.