Sequential model API in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Sequential model API

Problem:You are training a neural network to classify images into 10 categories using the Sequential model API in TensorFlow.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.85

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You must use the Sequential model API.

You can only change model architecture and training hyperparameters.

Do not change the dataset or preprocessing.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load dataset
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# Normalize pixel values
x_train, x_test = x_train / 255.0, x_test / 255.0

# Build Sequential model with dropout
model = models.Sequential([
    layers.Flatten(input_shape=(32, 32, 3)),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

# Compile model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train model with validation split and early stopping
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(x_train, y_train, epochs=30, batch_size=64, validation_split=0.2, callbacks=[early_stop])

# Evaluate on test data
test_loss, test_acc = model.evaluate(x_test, y_test)

print(f'Test accuracy: {test_acc:.2f}', f'Test loss: {test_loss:.2f}')

Added Dropout layers after dense layers to reduce overfitting.

Reduced the number of neurons in the second dense layer from 128 to 64.

Added EarlyStopping callback to stop training when validation loss stops improving.

Used a batch size of 64 and validation split of 0.2 for better generalization.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.85

After: Training accuracy: 90%, Validation accuracy: 86%, Training loss: 0.25, Validation loss: 0.40

Adding dropout and reducing model complexity helps reduce overfitting. Early stopping prevents training too long. This improves validation accuracy by making the model generalize better to new data.

Bonus Experiment

Try using batch normalization layers in the Sequential model to improve training stability and possibly increase validation accuracy.

💡 Hint

Insert batch normalization layers after dense layers and before activation functions.