Faster R-CNN usage in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Faster R-CNN usage

Which metric matters for Faster R-CNN and WHY

Faster R-CNN is used for object detection. The key metric is mean Average Precision (mAP). It measures how well the model finds objects and how accurate the bounding boxes are. mAP combines precision and recall over different object classes and detection thresholds. This helps us know if the model finds most objects (high recall) and if the found objects are correct (high precision).

Other useful metrics include Precision and Recall for each class, and Intersection over Union (IoU) to check how close predicted boxes are to real boxes.

Confusion matrix for object detection

Object detection does not use a simple confusion matrix like classification. Instead, it uses matches between predicted boxes and ground truth boxes based on IoU.

True Positives (TP): Predicted boxes correctly matched to ground truth (IoU >= threshold)
False Positives (FP): Predicted boxes with no matching ground truth
False Negatives (FN): Ground truth boxes missed by predictions
True Negatives (TN): Not usually defined in object detection

Example for one class:

Ground truth boxes: 10
Predicted boxes: 12
TP: 8
FP: 4 (12 - 8)
FN: 2 (10 - 8)

Precision = TP / (TP + FP) = 8 / 12 = 0.67
Recall = TP / (TP + FN) = 8 / 10 = 0.8

Precision vs Recall tradeoff with Faster R-CNN

In object detection, precision means how many detected objects are correct. Recall means how many real objects are found.

Example 1: High precision, low recall
The model only detects very clear objects, so almost all detections are correct (high precision). But it misses many objects (low recall).

Example 2: High recall, low precision
The model detects many objects, including uncertain ones. It finds most real objects (high recall) but also many wrong detections (low precision).

Depending on the use case, you may want to balance precision and recall. For example, in self-driving cars, missing an object (low recall) can be dangerous, so recall is very important.

What good vs bad metric values look like for Faster R-CNN

Good values:

• mAP > 0.7 means the model detects objects well with good accuracy.
• Precision and recall both above 0.7 show balanced detection.
• IoU threshold (e.g., 0.5) met consistently for bounding boxes.

Bad values:

• mAP below 0.3 means poor detection or many wrong boxes.
• Precision very low (< 0.5) means many false detections.
• Recall very low (< 0.5) means many missed objects.
• Bounding boxes with low IoU (< 0.3) mean inaccurate localization.

Common pitfalls in Faster R-CNN metrics

• Ignoring class imbalance: Some classes may have few examples, causing misleading high mAP if only common classes are detected well.
• Using accuracy: Accuracy is not useful for detection because many background areas exist and can inflate accuracy.
• Overfitting: Very high training mAP but low validation mAP means the model memorizes training data and won't generalize.
• Data leakage: If test images appear in training, metrics will be unrealistically high.
• Wrong IoU threshold: Using too low IoU threshold can make bounding boxes seem better than they are.

Self-check question

Your Faster R-CNN model has 98% accuracy but only 12% recall on detecting pedestrians. Is this good for production? Why or why not?

Answer: No, this is not good. Accuracy is misleading here because most image areas are background, so the model is mostly correct by saying "no pedestrian". But 12% recall means it misses 88% of pedestrians, which is dangerous for safety. You need to improve recall to catch more pedestrians.

Key Result

Mean Average Precision (mAP) is the key metric for Faster R-CNN, balancing precision and recall to measure detection quality.

Practice

1. What is the primary purpose of the Faster R-CNN model in PyTorch?

easy

A. To enhance image resolution

B. To generate captions for images

C. To detect objects and their locations in images

D. To classify images into categories without localization

Solution

1. Step 1: Understand Faster R-CNN's role

   Faster R-CNN is designed to find objects and their bounding boxes in images.

2. Step 2: Compare with other tasks

   Unlike captioning or classification without localization, Faster R-CNN outputs object locations and labels.

3. Final Answer:

   To detect objects and their locations in images -> Option C

4. Quick Check:

   Object detection = Faster R-CNN purpose [OK]

Hint: Faster R-CNN = object detection with bounding boxes [OK]

Common Mistakes:

• Confusing detection with classification only
• Thinking it generates image captions
• Assuming it enhances image quality

2. Which of the following is the correct way to load a pretrained Faster R-CNN model in PyTorch?

easy

A. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')

B. model = torchvision.models.fasterrcnn_resnet50_fpn(pretrained=False)

C. model = torchvision.models.fasterrcnn_resnet50_fpn(pretrained=True)

D. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights=None)

Solution

1. Step 1: Check PyTorch's current API for pretrained models

   PyTorch now uses the 'weights' argument with 'DEFAULT' to load pretrained weights.

2. Step 2: Identify correct syntax

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') uses weights='DEFAULT', which is the recommended way to load pretrained Faster R-CNN.

3. Final Answer:

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') -> Option A

4. Quick Check:

   Use weights='DEFAULT' for pretrained models [OK]

Hint: Use weights='DEFAULT' to load pretrained Faster R-CNN [OK]

Common Mistakes:

• Using deprecated pretrained=True argument
• Setting weights=None disables pretrained weights
• Confusing model namespaces in torchvision

3. Given the following code snippet, what is the output type of outputs after the model prediction?

import torchvision
import torch
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
image = torch.rand(3, 300, 400)  # Random image tensor
outputs = model([image])

medium

A. A list of dictionaries with keys 'boxes', 'labels', and 'scores'

B. A single tensor with class probabilities

C. A dictionary with keys 'boxes' and 'labels' only

D. A tensor of shape (3, 300, 400) identical to input

Solution

1. Step 1: Understand Faster R-CNN output format

   Faster R-CNN returns a list of dictionaries, one per input image, each containing 'boxes', 'labels', and 'scores'.

2. Step 2: Analyze the code snippet

   Since input is a list with one image, output is a list with one dictionary containing detection results.

3. Final Answer:

   A list of dictionaries with keys 'boxes', 'labels', and 'scores' -> Option A

4. Quick Check:

   Output = list of dicts with boxes, labels, scores [OK]

Hint: Faster R-CNN outputs list of dicts with boxes, labels, scores [OK]

Common Mistakes:

• Expecting a single tensor output
• Missing 'scores' key in output
• Confusing input and output tensor shapes

4. Identify the error in the following code snippet for using Faster R-CNN for prediction:

import torchvision
import torch
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
image = torch.rand(3, 224, 224)
outputs = model(image)

medium

A. Model.eval() should be called after prediction

B. Input image should be a list of tensors, not a single tensor

C. Weights argument 'DEFAULT' is invalid

D. Image tensor shape should be (224, 224, 3) instead of (3, 224, 224)

Solution

1. Step 1: Check input format for Faster R-CNN

   Faster R-CNN expects a list of image tensors, even if only one image is passed.

2. Step 2: Identify the error in the code

   The code passes a single tensor instead of a list, causing a runtime error.

3. Final Answer:

   Input image should be a list of tensors, not a single tensor -> Option B

4. Quick Check:

   Input = list of tensors for Faster R-CNN [OK]

Hint: Wrap input image in a list before passing to model [OK]

Common Mistakes:

• Passing single tensor instead of list
• Calling eval() after prediction
• Confusing channel order in tensor shape

5. You want to fine-tune a pretrained Faster R-CNN model on a custom dataset with 5 classes (including background). Which step is necessary before training?

hard

A. Change the input image size to (5, 224, 224)

B. Freeze all layers except the backbone

C. Set model.train() before loading pretrained weights

D. Replace the model's box predictor head with a new one having 5 output classes

Solution

1. Step 1: Understand model adaptation for new classes

   The box predictor head controls class outputs; it must match the number of classes in your dataset.

2. Step 2: Replace the box predictor head

   Replace the existing head with a new one configured for 5 classes to fine-tune properly.

3. Final Answer:

   Replace the model's box predictor head with a new one having 5 output classes -> Option D

4. Quick Check:

   Adjust box predictor for new class count [OK]

Hint: Change box predictor head to match your classes [OK]

Common Mistakes:

• Changing input image channels incorrectly
• Not replacing box predictor head
• Freezing all layers prevents fine-tuning