Experiment - Model evaluation best practices

Problem:You have trained a computer vision model to classify images into 5 categories. The model shows 95% accuracy on training data but only 70% accuracy on validation data.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Validation loss: 1.2

Issue:The model is overfitting. It performs very well on training data but poorly on validation data, indicating it does not generalize well.

Your Task

Reduce overfitting and improve validation accuracy to at least 85% while keeping training accuracy below 90%.

You can only modify model evaluation and training best practices, not the model architecture.

Do not add new data or change the dataset.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.metrics import classification_report, confusion_matrix

# Assume X_train, y_train, X_val, y_val are preloaded image data and labels

# Data augmentation setup
train_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

val_datagen = ImageDataGenerator()

train_generator = train_datagen.flow(X_train, y_train, batch_size=32)
val_generator = val_datagen.flow(X_val, y_val, batch_size=32, shuffle=False)

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Assume model is predefined
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = model.fit(
    train_generator,
    epochs=50,
    validation_data=val_generator,
    callbacks=[early_stop]
)

# Evaluate on validation data
val_loss, val_accuracy = model.evaluate(val_generator)

# Predict on validation data
y_pred_probs = model.predict(val_generator)
y_pred = y_pred_probs.argmax(axis=1)

# Print classification report and confusion matrix
print(classification_report(y_val, y_pred))
print(confusion_matrix(y_val, y_pred))

Added data augmentation to training data to improve model generalization.

Implemented early stopping to stop training when validation loss stops improving.

Used proper validation data generator without augmentation for evaluation.

Evaluated model with classification report and confusion matrix for detailed metrics.

Results Interpretation

Before: Training accuracy: 95%, Validation accuracy: 70%, Validation loss: 1.2

After: Training accuracy: 88%, Validation accuracy: 86%, Validation loss: 0.6

Using best practices like data augmentation and early stopping helps reduce overfitting, improving validation accuracy and model generalization.

Bonus Experiment

Try using k-fold cross-validation to evaluate the model more reliably across different data splits.

💡 Hint

Use sklearn's KFold to split data and train the model multiple times, then average the validation metrics.