Experiment - Why detection localizes objects in images

Problem:We want to teach a model to find and draw boxes around objects in pictures. The current model can tell if an object is there but does not draw boxes accurately.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Localization error (IoU < 0.5): 60%

Issue:The model overfits classification but fails to localize objects well, meaning it detects objects but the boxes are often wrong or misplaced.

Your Task

Improve the model so it localizes objects better, aiming for localization error below 30% and validation accuracy above 80%.

Keep the same dataset and model architecture base (e.g., Faster R-CNN).

Only adjust training hyperparameters and add regularization techniques.

Do not change the backbone network.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras.layers import Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Assume base_model is a Faster R-CNN like model already defined
# Add dropout to classification and box regression heads

class DetectionModel(tf.keras.Model):
    def __init__(self, base_model):
        super().__init__()
        self.base_model = base_model
        self.dropout = Dropout(0.3)

    def call(self, inputs, training=False):
        features = self.base_model(inputs, training=training)
        if training:
            features = self.dropout(features, training=training)
        return features

# Data augmentation
train_datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.1,
    height_shift_range=0.1,
    horizontal_flip=True
)

# Compile model with lower learning rate
model = DetectionModel(base_model)
optimizer = Adam(learning_rate=0.0001)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

# Train model with augmented data
model.fit(
    train_datagen.flow(X_train, y_train, batch_size=32),
    epochs=20,
    validation_data=(X_val, y_val)
)

# Evaluate localization with IoU metric (pseudo code)
def compute_iou(boxA, boxB):
    # Calculate intersection
    xA = max(boxA[0], boxB[0])
    yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2])
    yB = min(boxA[3], boxB[3])
    interArea = max(0, xB - xA) * max(0, yB - yA)
    # Calculate union
    boxAArea = (boxA[2] - boxA[0]) * (boxA[3] - boxA[1])
    boxBArea = (boxB[2] - boxB[0]) * (boxB[3] - boxB[1])
    iou = interArea / float(boxAArea + boxBArea - interArea)
    return iou

# After training, compute IoU on validation set to check localization improvement

Added dropout layers to reduce overfitting in detection heads.

Applied data augmentation to increase training data variety.

Reduced learning rate for more stable training.

Monitored IoU metric to evaluate localization quality.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Localization error 60%

After: Training accuracy 90%, Validation accuracy 83%, Localization error 25%

Adding dropout and data augmentation helped the model generalize better, reducing overfitting and improving its ability to draw accurate boxes around objects. This shows that localization needs both good classification and precise bounding box prediction, which benefit from regularization and diverse training data.

Bonus Experiment

Try using a different backbone network like MobileNetV2 to see if a lighter model can still localize objects well.

💡 Hint

MobileNetV2 is faster and smaller; fine-tune it with the same dropout and augmentation to compare localization accuracy and speed.