Multi-label classification in ML Python - ML Experiment: Train & Evaluate

Experiment - Multi-label classification

Problem:We want to classify images where each image can have multiple labels at the same time, like tagging a photo with 'beach', 'sunset', and 'people'. Our current model predicts labels but tends to overfit the training data.

Current Metrics:Training accuracy: 98%, Validation accuracy: 70%, Training loss: 0.05, Validation loss: 0.45

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only change the model architecture and training hyperparameters.

Do not change the dataset or data preprocessing steps.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preloaded numpy arrays

model = Sequential([
    Dense(128, activation='relu', input_shape=(X_train.shape[1],)),
    BatchNormalization(),
    Dropout(0.4),
    Dense(64, activation='relu'),
    BatchNormalization(),
    Dropout(0.3),
    Dense(y_train.shape[1], activation='sigmoid')  # sigmoid for multi-label
])

model.compile(optimizer=Adam(learning_rate=0.001),
              loss='binary_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=50,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop],
                    verbose=0)

train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
train_loss = history.history['loss'][-1]
val_loss = history.history['val_loss'][-1]

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')
print(f'Training loss: {train_loss:.4f}')
print(f'Validation loss: {val_loss:.4f}')

Added Dropout layers after dense layers to reduce overfitting by randomly turning off neurons during training.

Added BatchNormalization layers to stabilize and speed up training.

Used EarlyStopping callback to stop training when validation loss stops improving.

Reduced learning rate to 0.001 for smoother training.

Results Interpretation

Before: Training accuracy was 98%, validation accuracy was 70%. The large gap shows overfitting.

After: Training accuracy dropped to 90%, validation accuracy improved to 87%. Loss values also show better generalization.

Adding dropout and batch normalization, along with early stopping and a lower learning rate, helps reduce overfitting and improves the model's ability to generalize to new data in multi-label classification.

Bonus Experiment

Try using a convolutional neural network (CNN) architecture instead of a simple dense network for multi-label classification on image data.

💡 Hint

CNNs can better capture spatial features in images, which often improves multi-label classification performance.