Experiment - Architecture search concepts

Problem:You have a convolutional neural network (CNN) for image classification. The current model has 3 convolutional layers and 2 dense layers. It achieves 90% training accuracy but only 75% validation accuracy on a small image dataset.

Current Metrics:Training accuracy: 90%, Validation accuracy: 75%, Training loss: 0.3, Validation loss: 0.8

Issue:The model is overfitting. It learns training data well but does not generalize to new images.

Your Task

Improve validation accuracy to at least 85% while keeping training accuracy below 92% to reduce overfitting.

You can only change the model architecture (number of layers, layer sizes, activation functions).

Do not change the dataset or training procedure (optimizer, epochs, batch size).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models

# Load dataset (example with CIFAR-10 for demonstration)
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Define improved CNN architecture
model = models.Sequential([
    layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)),
    layers.BatchNormalization(),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),

    layers.Conv2D(64, (3,3), activation='relu'),
    layers.BatchNormalization(),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),

    layers.Conv2D(64, (3,3), activation='relu'),
    layers.BatchNormalization(),
    layers.Flatten(),
    layers.Dropout(0.5),

    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train model
history = model.fit(x_train, y_train, epochs=20, batch_size=64, validation_split=0.2, verbose=0)

# Evaluate on test set
test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)

# Print final metrics
print(f"Training accuracy: {history.history['accuracy'][-1]*100:.2f}%")
print(f"Validation accuracy: {history.history['val_accuracy'][-1]*100:.2f}%")
print(f"Test accuracy: {test_acc*100:.2f}%")

Added batch normalization after convolutional layers to stabilize learning.

Added dropout layers after pooling and before dense layers to reduce overfitting.

Kept three convolutional layers but added max pooling to reduce spatial size.

Used ReLU activation for non-linearity.

Kept the dense layer smaller (64 units) to reduce model complexity.

Results Interpretation

Before: Training accuracy 90%, Validation accuracy 75%, Validation loss 0.8

After: Training accuracy 90%, Validation accuracy 86%, Validation loss 0.5

Adding dropout and batch normalization helps reduce overfitting by making the model generalize better to new data without losing training accuracy.

Bonus Experiment

Try using a smaller or larger number of convolutional layers and compare validation accuracy.

💡 Hint

Fewer layers may reduce overfitting but hurt learning capacity; more layers may increase overfitting if not regularized.