torchvision detection models in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - torchvision detection models

Which metric matters for torchvision detection models and WHY

For object detection models like those in torchvision, the key metric is Mean Average Precision (mAP). This metric measures how well the model finds and correctly labels objects in images. It balances both precision (how many detected objects are correct) and recall (how many real objects are found). Since detection involves locating objects and classifying them, mAP gives a clear picture of overall performance.

Other useful metrics include Precision and Recall at different Intersection over Union (IoU) thresholds, which show how strict the model is about matching predicted boxes to real boxes.

Confusion matrix or equivalent visualization

In object detection, confusion matrices are less straightforward because predictions include bounding boxes and classes. Instead, we use a table of True Positives (TP), False Positives (FP), and False Negatives (FN) based on IoU threshold matching.

Class: Cat
+----------------+----------------+----------------+
|                | Predicted Cat  | Predicted Not  |
|                |                | Cat            |
+----------------+----------------+----------------+
| Actual Cat     | TP = 80        | FN = 20        |
+----------------+----------------+----------------+
| Actual Not Cat | FP = 15        | TN = (ignored) |
+----------------+----------------+----------------+

Note: TN (True Negative) is not usually counted in detection metrics because the background is large and undefined.
    

Precision vs Recall tradeoff with concrete examples

Imagine a security camera detecting people:

• High Precision, Low Recall: The model only reports people when very sure. It misses some people (low recall) but rarely mistakes objects for people (high precision). Good if false alarms are costly.
• High Recall, Low Precision: The model tries to find every person, even if unsure. It finds almost all people (high recall) but sometimes mistakes objects for people (low precision). Good if missing a person is worse than false alarms.

mAP balances these by averaging precision over recall levels and IoU thresholds.

What "good" vs "bad" metric values look like for torchvision detection models

For mAP (usually reported as a percentage):

• Good: mAP above 50% on challenging datasets like COCO means the model detects and classifies objects well.
• Bad: mAP below 20% means the model struggles to find or correctly label objects.

Precision and recall values closer to 1.0 (or 100%) are better, but often there is a tradeoff.

Common pitfalls in metrics for detection models

• Ignoring IoU thresholds: Counting predictions as correct without checking if bounding boxes overlap enough can inflate metrics.
• Data leakage: Testing on images the model saw during training gives unrealistically high scores.
• Overfitting: Very high training mAP but low test mAP means the model memorizes training images but fails to generalize.
• Ignoring class imbalance: Some classes may be rare, so overall mAP can hide poor performance on those classes.

Self-check question

Your torchvision detection model has 98% accuracy but only 12% recall on detecting cars. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because it likely counts many true negatives (background). The very low recall means the model misses most cars, which is bad if detecting cars is important. You want both high precision and recall for reliable detection.

Key Result

Mean Average Precision (mAP) is the key metric for torchvision detection models, balancing precision and recall over object localization and classification.

Practice

1. Which of the following is a detection model provided by torchvision for finding objects in images?

easy

A. ResNet

B. Faster R-CNN

C. VGG16

D. Transformer

Solution

1. Step 1: Understand torchvision detection models

   torchvision offers models specifically for object detection like Faster R-CNN, Mask R-CNN, and RetinaNet.

2. Step 2: Identify the correct model

   ResNet and VGG16 are classification models, and Transformer is a general architecture, not a torchvision detection model.

3. Final Answer:

   Faster R-CNN -> Option B

4. Quick Check:

   Detection model = Faster R-CNN [OK]

Hint: Detection models include 'R-CNN' names in torchvision [OK]

Common Mistakes:

• Confusing classification models with detection models
• Thinking Transformer is a torchvision detection model
• Mixing up model types

2. Which code snippet correctly loads a pretrained Faster R-CNN model from torchvision?

easy

A. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')

B. model = torchvision.models.fasterrcnn_resnet50_fpn(pretrained=True)

C. model = torchvision.detection.fasterrcnn_resnet50_fpn(pretrained=True)

D. model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)

Solution

1. Step 1: Check the correct module and argument

   In recent torchvision versions, detection models are under torchvision.models.detection and use weights='DEFAULT' instead of pretrained=True.

2. Step 2: Identify the correct syntax

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') uses the correct module path and argument for loading pretrained weights.

3. Final Answer:

   model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT') -> Option A

4. Quick Check:

   Use weights='DEFAULT' to load pretrained detection models [OK]

Hint: Use weights='DEFAULT' for pretrained detection models [OK]

Common Mistakes:

• Using pretrained=True instead of weights='DEFAULT'
• Wrong module path
• Missing 'models' in import path

3. What will be the output type of the following code snippet?

import torch
import torchvision
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
inputs = [torch.rand(3, 300, 400)]
outputs = model(inputs)
print(type(outputs))

medium

A.

B.

C.

D.

Solution

1. Step 1: Understand model output format

   torchvision detection models return a list of dictionaries, one per input image.

2. Step 2: Check the type of outputs variable

   Since inputs is a list with one tensor, outputs will be a list containing one dictionary.

3. Final Answer:

   <class 'list'> -> Option D

4. Quick Check:

   Detection model output = list of dicts [OK]

Hint: Detection models output a list, one dict per input image [OK]

Common Mistakes:

• Expecting a single tensor output
• Confusing output with dict directly
• Not wrapping inputs in a list

4. Identify the error in this code snippet that tries to use a pretrained Mask R-CNN model:

import torchvision
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
model.eval()

medium

A. The function name should be fasterrcnn_resnet50_fpn

B. Mask R-CNN is not available in torchvision

C. The argument 'pretrained' is deprecated; use 'weights' instead

D. Missing import for torch

Solution

1. Step 1: Check function and argument usage

   Mask R-CNN is available as maskrcnn_resnet50_fpn in torchvision.models.detection.

2. Step 2: Identify deprecated argument

   Using pretrained=True is deprecated; the correct way is weights='DEFAULT'.

3. Final Answer:

   The argument 'pretrained' is deprecated; use 'weights' instead -> Option C

4. Quick Check:

   Use weights='DEFAULT' instead of pretrained=True [OK]

Hint: Replace pretrained=True with weights='DEFAULT' [OK]

Common Mistakes:

• Using deprecated pretrained argument
• Confusing model names
• Assuming Mask R-CNN is missing

5. You want to fine-tune a pretrained RetinaNet model from torchvision on your custom dataset with 5 classes. Which step is necessary before training?

hard

A. Replace the model's box predictor head to output 6 classes (5 + background)

B. Change the input image size to 224x224

C. Set pretrained=False to train from scratch

D. Freeze all layers except the backbone

Solution

1. Step 1: Understand class output adjustment

   Detection models include a background class, so for 5 classes, output head must predict 6 classes.

2. Step 2: Modify the box predictor head

   Replace the box predictor to match the number of classes (5 + 1 background) before training.

3. Final Answer:

   Replace the model's box predictor head to output 6 classes (5 + background) -> Option A

4. Quick Check:

   Adjust output head for custom classes [OK]

Hint: Add 1 for background class when changing output head [OK]

Common Mistakes:

• Not adjusting output head for new classes
• Changing input size unnecessarily
• Freezing all layers without fine-tuning head