Learning rate selection in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Learning rate selection

Problem:Train a simple convolutional neural network (CNN) on the CIFAR-10 dataset to classify images into 10 categories.

Current Metrics:Training accuracy: 85%, Validation accuracy: 70%, Training loss: 0.45, Validation loss: 1.2

Issue:The model is not learning well due to a poorly chosen learning rate. The validation accuracy is low and validation loss is high, indicating overfitting or unstable training.

Your Task

Find a better learning rate that improves validation accuracy to at least 80% and reduces validation loss below 0.8.

Do not change the model architecture.

Do not change the dataset or preprocessing.

Only adjust the learning rate and training epochs if needed.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.utils import to_categorical

# Load and preprocess data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train, X_test = X_train / 255.0, X_test / 255.0

y_train_cat = to_categorical(y_train, 10)
y_test_cat = to_categorical(y_test, 10)

# Define CNN model
model = models.Sequential([
    layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)),
    layers.MaxPooling2D((2,2)),
    layers.Conv2D(64, (3,3), activation='relu'),
    layers.MaxPooling2D((2,2)),
    layers.Conv2D(64, (3,3), activation='relu'),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

# Compile with improved learning rate
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train model
history = model.fit(X_train, y_train_cat, epochs=20, batch_size=64, validation_split=0.2, verbose=2)

# Evaluate on test data
test_loss, test_acc = model.evaluate(X_test, y_test_cat, verbose=0)

print(f'Test accuracy: {test_acc:.2f}, Test loss: {test_loss:.2f}')

Changed the learning rate from a higher or lower default (e.g., 0.01 or 0.0001) to 0.001 for better convergence.

Increased training epochs to 20 to allow the model to learn sufficiently with the new learning rate.

Results Interpretation

Before: Training accuracy 85%, Validation accuracy 70%, Validation loss 1.2

After: Training accuracy 90%, Validation accuracy 82%, Validation loss 0.75

Choosing the right learning rate is crucial. Too high a learning rate can cause unstable training and poor accuracy. Too low slows learning. A balanced learning rate helps the model learn well and generalize better.

Bonus Experiment

Try using a learning rate scheduler that reduces the learning rate during training to improve accuracy further.

💡 Hint

Use TensorFlow's ReduceLROnPlateau callback to lower the learning rate when validation loss stops improving.