torchvision pre-trained models in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - torchvision pre-trained models

Which metric matters for torchvision pre-trained models and WHY

When using torchvision pre-trained models, the key metrics depend on the task. For image classification, accuracy is often used to measure how many images are correctly labeled. However, accuracy alone can be misleading if classes are unbalanced.

For more detailed insight, precision, recall, and F1 score help understand how well the model identifies each class, especially for rare classes. For example, in medical image classification, recall is critical to catch all positive cases.

In object detection or segmentation tasks, metrics like mean Average Precision (mAP) or Intersection over Union (IoU) are used, but for simple classification, accuracy, precision, recall, and F1 are the main metrics.

Confusion matrix example

Suppose a pre-trained model classifies 100 images into two classes: Cat and Dog.

      | Predicted Cat | Predicted Dog |
      |--------------|---------------|
      | True Cat: 40 | False Dog: 5  |
      | False Cat: 10| True Dog: 45  |
    

Here:

• True Positives (TP) for Cat = 40
• False Positives (FP) for Cat = 10
• False Negatives (FN) for Cat = 5
• True Negatives (TN) for Cat = 45

From this, precision for Cat = 40 / (40 + 10) = 0.80, recall for Cat = 40 / (40 + 5) = 0.89.

Precision vs Recall tradeoff with examples

Using a torchvision pre-trained model, you might adjust thresholds to favor precision or recall.

• High Precision: The model is very sure when it says an image is a Cat. This reduces false alarms (wrongly labeling Dogs as Cats). Useful when false alarms are costly, like in security.
• High Recall: The model finds almost all Cats, even if some Dogs are mislabeled. Useful when missing a Cat is worse, like detecting rare diseases in images.

Choosing the right balance depends on your goal and the cost of mistakes.

What "good" vs "bad" metric values look like for torchvision pre-trained models

Good metrics for a well-performing pre-trained model on a balanced classification task might be:

• Accuracy > 85%
• Precision > 80%
• Recall > 80%
• F1 score > 80%

Bad metrics might be:

• Accuracy < 60%
• Precision or Recall < 50%
• F1 score < 50%

However, these numbers depend on the dataset and task difficulty.

Common pitfalls when evaluating torchvision pre-trained models

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% images are Dogs, predicting Dog always gives 90% accuracy but no real learning.
• Data leakage: Using test images during training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data but fails to generalize.
• Ignoring class imbalance: Not using precision, recall, or F1 can hide poor performance on rare classes.

Self-check question

Your torchvision pre-trained model has 98% accuracy but only 12% recall on the rare class (e.g., fraud images). Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of the rare class cases, which is critical if catching those cases is important. High accuracy is misleading because the rare class is small. You need to improve recall to catch more rare cases.

Key Result

Accuracy alone can be misleading; precision, recall, and F1 score provide a clearer picture of pre-trained model performance.