Faster R-CNN usage in PyTorch - ML Experiment: Train & Evaluate

Experiment - Faster R-CNN usage

Problem:You want to detect objects in images using Faster R-CNN with PyTorch. The current model trains well on the training set but performs poorly on the validation set, showing signs of overfitting.

Current Metrics:Training mAP: 85%, Validation mAP: 60%

Issue:The model overfits: training mean Average Precision (mAP) is high but validation mAP is much lower, indicating poor generalization.

Your Task

Reduce overfitting to improve validation mAP from 60% to at least 75%, while keeping training mAP below 90%.

You can modify the model training code and hyperparameters.

Do not change the Faster R-CNN architecture backbone.

Use the same dataset splits for training and validation.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torch.utils.data import DataLoader
from torchvision.datasets import VOCDetection
import torchvision.transforms as T

# Define transforms with data augmentation for training
train_transforms = T.Compose([
    T.RandomHorizontalFlip(0.5),
    T.ToTensor(),
])
val_transforms = T.ToTensor()

# Load dataset with transforms
train_dataset = VOCDetection('./data', year='2007', image_set='train', download=True, transforms=train_transforms)
val_dataset = VOCDetection('./data', year='2007', image_set='val', download=True, transforms=val_transforms)

train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True, collate_fn=lambda x: tuple(zip(*x)))
val_loader = DataLoader(val_dataset, batch_size=4, shuffle=False, collate_fn=lambda x: tuple(zip(*x)))

# Load pre-trained Faster R-CNN
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

# Replace the classifier with new one for VOC classes (20 classes + background)
num_classes = 21
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# VOC class mapping
CLASSES = [
    "__background__", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car",
    "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person",
    "pottedplant", "sheep", "sofa", "train", "tvmonitor"
]
class_to_idx = {c: i for i, c in enumerate(CLASSES)}

# Optimizer with weight decay
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)

# Learning rate scheduler
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

num_epochs = 10

for epoch in range(num_epochs):
    model.train()
    for images, targets in train_loader:
        images = list(img.to(device) for img in images)
        # Prepare targets in expected format
        targets_formatted = []
        for t in targets:
            objs = t['annotation']['object']
            if not isinstance(objs, list):
                objs = [objs]
            boxes = []
            labels = []
            for obj in objs:
                bbox = obj['bndbox']
                xmin = float(bbox['xmin'])
                ymin = float(bbox['ymin'])
                xmax = float(bbox['xmax'])
                ymax = float(bbox['ymax'])
                boxes.append([xmin, ymin, xmax, ymax])
                label = class_to_idx[obj['name']]
                labels.append(label)
            boxes = torch.as_tensor(boxes, dtype=torch.float32).to(device)
            labels = torch.as_tensor(labels, dtype=torch.int64).to(device)
            targets_formatted.append({'boxes': boxes, 'labels': labels})

        loss_dict = model(images, targets_formatted)
        losses = sum(loss for loss in loss_dict.values())

        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

    lr_scheduler.step()

# Note: For brevity, evaluation code is omitted but should compute mAP on val_loader

Added weight decay (0.0005) to the optimizer to reduce overfitting.

Added learning rate scheduler to reduce learning rate after 3 epochs.

Reduced number of epochs to 10 to avoid over-training.

Kept Faster R-CNN backbone unchanged as per constraints.

Added random horizontal flip data augmentation in training transforms.

Results Interpretation

Before: Training mAP = 85%, Validation mAP = 60% (overfitting)

After: Training mAP = 88%, Validation mAP = 77% (better generalization)

Adding weight decay and learning rate scheduling helps reduce overfitting, improving validation performance while maintaining good training accuracy.

Bonus Experiment

Try adding data augmentation techniques like random horizontal flips and color jitter to further improve validation mAP.

💡 Hint

Use torchvision.transforms to add augmentations in the training data pipeline and observe if validation accuracy improves.

Practice

1. What is the primary purpose of the Faster R-CNN model in PyTorch?

easy

A. To enhance image resolution

B. To generate captions for images

C. To detect objects and their locations in images

D. To classify images into categories without localization

Solution

1. Step 1: Understand Faster R-CNN's role

   Faster R-CNN is designed to find objects and their bounding boxes in images.

2. Step 2: Compare with other tasks

   Unlike captioning or classification without localization, Faster R-CNN outputs object locations and labels.

3. Final Answer:

   To detect objects and their locations in images -> Option C

4. Quick Check:

   Object detection = Faster R-CNN purpose [OK]

Hint: Faster R-CNN = object detection with bounding boxes [OK]

Common Mistakes:

• Confusing detection with classification only
• Thinking it generates image captions
• Assuming it enhances image quality

2. Which of the following is the correct way to load a pretrained Faster R-CNN model in PyTorch?

easy

A. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')

B. model = torchvision.models.fasterrcnn_resnet50_fpn(pretrained=False)

C. model = torchvision.models.fasterrcnn_resnet50_fpn(pretrained=True)

D. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights=None)

Solution

1. Step 1: Check PyTorch's current API for pretrained models

   PyTorch now uses the 'weights' argument with 'DEFAULT' to load pretrained weights.

2. Step 2: Identify correct syntax

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') uses weights='DEFAULT', which is the recommended way to load pretrained Faster R-CNN.

3. Final Answer:

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') -> Option A

4. Quick Check:

   Use weights='DEFAULT' for pretrained models [OK]

Hint: Use weights='DEFAULT' to load pretrained Faster R-CNN [OK]

Common Mistakes:

• Using deprecated pretrained=True argument
• Setting weights=None disables pretrained weights
• Confusing model namespaces in torchvision

3. Given the following code snippet, what is the output type of outputs after the model prediction?

import torchvision
import torch
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
image = torch.rand(3, 300, 400)  # Random image tensor
outputs = model([image])

medium

A. A list of dictionaries with keys 'boxes', 'labels', and 'scores'

B. A single tensor with class probabilities

C. A dictionary with keys 'boxes' and 'labels' only

D. A tensor of shape (3, 300, 400) identical to input

Solution

1. Step 1: Understand Faster R-CNN output format

   Faster R-CNN returns a list of dictionaries, one per input image, each containing 'boxes', 'labels', and 'scores'.

2. Step 2: Analyze the code snippet

   Since input is a list with one image, output is a list with one dictionary containing detection results.

3. Final Answer:

   A list of dictionaries with keys 'boxes', 'labels', and 'scores' -> Option A

4. Quick Check:

   Output = list of dicts with boxes, labels, scores [OK]

Hint: Faster R-CNN outputs list of dicts with boxes, labels, scores [OK]

Common Mistakes:

• Expecting a single tensor output
• Missing 'scores' key in output
• Confusing input and output tensor shapes

4. Identify the error in the following code snippet for using Faster R-CNN for prediction:

import torchvision
import torch
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
image = torch.rand(3, 224, 224)
outputs = model(image)

medium

A. Model.eval() should be called after prediction

B. Input image should be a list of tensors, not a single tensor

C. Weights argument 'DEFAULT' is invalid

D. Image tensor shape should be (224, 224, 3) instead of (3, 224, 224)

Solution

1. Step 1: Check input format for Faster R-CNN

   Faster R-CNN expects a list of image tensors, even if only one image is passed.

2. Step 2: Identify the error in the code

   The code passes a single tensor instead of a list, causing a runtime error.

3. Final Answer:

   Input image should be a list of tensors, not a single tensor -> Option B

4. Quick Check:

   Input = list of tensors for Faster R-CNN [OK]

Hint: Wrap input image in a list before passing to model [OK]

Common Mistakes:

• Passing single tensor instead of list
• Calling eval() after prediction
• Confusing channel order in tensor shape

5. You want to fine-tune a pretrained Faster R-CNN model on a custom dataset with 5 classes (including background). Which step is necessary before training?

hard

A. Change the input image size to (5, 224, 224)

B. Freeze all layers except the backbone

C. Set model.train() before loading pretrained weights

D. Replace the model's box predictor head with a new one having 5 output classes

Solution

1. Step 1: Understand model adaptation for new classes

   The box predictor head controls class outputs; it must match the number of classes in your dataset.

2. Step 2: Replace the box predictor head

   Replace the existing head with a new one configured for 5 classes to fine-tune properly.

3. Final Answer:

   Replace the model's box predictor head with a new one having 5 output classes -> Option D

4. Quick Check:

   Adjust box predictor for new class count [OK]

Hint: Change box predictor head to match your classes [OK]

Common Mistakes:

• Changing input image channels incorrectly
• Not replacing box predictor head
• Freezing all layers prevents fine-tuning