Backpropagation concept in ML Python - ML Experiment: Train & Evaluate

Experiment - Backpropagation concept

Problem:Train a simple neural network to classify handwritten digits using backpropagation.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting: training accuracy is high but validation accuracy is much lower.

Your Task

Reduce overfitting by improving validation accuracy to at least 85% while keeping training accuracy below 92%.

Keep the same neural network architecture (one hidden layer with 128 neurons).

Do not change the dataset or input features.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import tensorflow as tf
from tensorflow.keras import layers, models, regularizers

# Load dataset
mnist = tf.keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalize data
X_train, X_test = X_train / 255.0, X_test / 255.0

# Build model with dropout and L2 regularization
model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

# Compile model with lower learning rate
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=optimizer,
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train model
history = model.fit(X_train, y_train, epochs=20, batch_size=64, validation_split=0.2, verbose=0)

# Evaluate model
train_loss, train_acc = model.evaluate(X_train, y_train, verbose=0)
val_loss, val_acc = model.evaluate(X_test, y_test, verbose=0)

print(f'Training accuracy: {train_acc*100:.2f}%, Validation accuracy: {val_acc*100:.2f}%')
print(f'Training loss: {train_loss:.3f}, Validation loss: {val_loss:.3f}')

Added dropout layer with rate 0.3 after the hidden layer to reduce overfitting.

Applied L2 regularization with factor 0.001 to the hidden layer weights.

Reduced learning rate from default to 0.001 for smoother training.

Results Interpretation

Before: Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

After: Training accuracy: 90.5%, Validation accuracy: 86.7%, Training loss: 0.28, Validation loss: 0.35

Adding dropout and L2 regularization helps reduce overfitting by preventing the model from relying too much on specific neurons or large weights. Lowering the learning rate helps the model learn more steadily. This improves validation accuracy while slightly lowering training accuracy, showing better generalization.

Bonus Experiment

Try using batch normalization instead of dropout and compare the validation accuracy.

💡 Hint

Add a BatchNormalization layer after the hidden Dense layer and remove dropout. Observe if the model trains faster and generalizes well.