torchvision pre-trained models in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - torchvision pre-trained models

Which metric matters for torchvision pre-trained models and WHY

When using torchvision pre-trained models, the key metrics depend on the task. For image classification, accuracy is often used to measure how many images are correctly labeled. However, accuracy alone can be misleading if classes are unbalanced.

For more detailed insight, precision, recall, and F1 score help understand how well the model identifies each class, especially for rare classes. For example, in medical image classification, recall is critical to catch all positive cases.

In object detection or segmentation tasks, metrics like mean Average Precision (mAP) or Intersection over Union (IoU) are used, but for simple classification, accuracy, precision, recall, and F1 are the main metrics.

Confusion matrix example

Suppose a pre-trained model classifies 100 images into two classes: Cat and Dog.

      | Predicted Cat | Predicted Dog |
      |--------------|---------------|
      | True Cat: 40 | False Dog: 5  |
      | False Cat: 10| True Dog: 45  |
    

Here:

• True Positives (TP) for Cat = 40
• False Positives (FP) for Cat = 10
• False Negatives (FN) for Cat = 5
• True Negatives (TN) for Cat = 45

From this, precision for Cat = 40 / (40 + 10) = 0.80, recall for Cat = 40 / (40 + 5) = 0.89.

Precision vs Recall tradeoff with examples

Using a torchvision pre-trained model, you might adjust thresholds to favor precision or recall.

• High Precision: The model is very sure when it says an image is a Cat. This reduces false alarms (wrongly labeling Dogs as Cats). Useful when false alarms are costly, like in security.
• High Recall: The model finds almost all Cats, even if some Dogs are mislabeled. Useful when missing a Cat is worse, like detecting rare diseases in images.

Choosing the right balance depends on your goal and the cost of mistakes.

What "good" vs "bad" metric values look like for torchvision pre-trained models

Good metrics for a well-performing pre-trained model on a balanced classification task might be:

• Accuracy > 85%
• Precision > 80%
• Recall > 80%
• F1 score > 80%

Bad metrics might be:

• Accuracy < 60%
• Precision or Recall < 50%
• F1 score < 50%

However, these numbers depend on the dataset and task difficulty.

Common pitfalls when evaluating torchvision pre-trained models

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% images are Dogs, predicting Dog always gives 90% accuracy but no real learning.
• Data leakage: Using test images during training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data but fails to generalize.
• Ignoring class imbalance: Not using precision, recall, or F1 can hide poor performance on rare classes.

Self-check question

Your torchvision pre-trained model has 98% accuracy but only 12% recall on the rare class (e.g., fraud images). Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of the rare class cases, which is critical if catching those cases is important. High accuracy is misleading because the rare class is small. You need to improve recall to catch more rare cases.

Key Result

Accuracy alone can be misleading; precision, recall, and F1 score provide a clearer picture of pre-trained model performance.

Practice

1. What is the main advantage of using torchvision pre-trained models?

easy

A. They automatically improve your dataset quality.

B. They generate new images from text descriptions.

C. They reduce the size of your images.

D. They allow you to use powerful image models without training from scratch.

Solution

1. Step 1: Understand what pre-trained models do

   Pre-trained models are already trained on large datasets, so you don't need to train them from zero.

2. Step 2: Identify the main benefit

   This saves time and resources, letting you use powerful models quickly.

3. Final Answer:

   They allow you to use powerful image models without training from scratch. -> Option D

4. Quick Check:

   Pre-trained models = reuse trained weights [OK]

Hint: Pre-trained means ready to use without full training [OK]

Common Mistakes:

• Thinking pre-trained models improve data quality
• Confusing pre-trained models with image resizing
• Believing they generate images from text

2. Which of the following is the correct way to load a pre-trained ResNet18 model from torchvision?

easy

A. model = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)

B. model = torchvision.resnet18(pretrained=True)

C. model = torchvision.models.resnet18(pretrained=False)

D. model = torchvision.models.load_resnet18(pretrained=True)

Solution

1. Step 1: Recall the updated torchvision syntax

   Since torchvision 0.13+, pre-trained weights are loaded using the 'weights' argument with a weights enum, not 'pretrained=True'.

2. Step 2: Identify the correct syntax for ResNet18

   Use torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1).

3. Final Answer:

   model = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1) -> Option A

4. Quick Check:

   Use weights=enum, not pretrained=True [OK]

Hint: Use weights= argument, not pretrained=True [OK]

Common Mistakes:

• Using pretrained=False which doesn't load pre-trained weights
• Calling torchvision.resnet18 directly
• Using a non-existent load_resnet18 function

3. What will be the output shape of the following code snippet using a pre-trained ResNet18 model on a batch of 8 RGB images of size 224x224?

import torch
import torchvision.models as models
model = models.resnet18(weights=models.ResNet18_Weights.IMAGENET1K_V1)
model.eval()
inputs = torch.randn(8, 3, 224, 224)
outputs = model(inputs)
print(outputs.shape)

medium

A. torch.Size([8, 3, 224, 224])

B. torch.Size([8, 1000])

C. torch.Size([8, 512])

D. torch.Size([1, 1000])

Solution

1. Step 1: Understand ResNet18 output size

   ResNet18 pre-trained on ImageNet outputs logits for 1000 classes, so output shape is (batch_size, 1000).

2. Step 2: Check input batch size and output shape

   Input batch size is 8, so output shape is (8, 1000).

3. Final Answer:

   torch.Size([8, 1000]) -> Option B

4. Quick Check:

   Batch size 8, 1000 classes output [OK]

Hint: Output shape = (batch_size, number_of_classes) [OK]

Common Mistakes:

• Confusing output with input image shape
• Expecting feature vector size instead of class logits
• Assuming batch size 1 output

4. You wrote this code to use a pre-trained model for prediction but get wrong results:

model = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.IMAGENET1K_V1)
inputs = torch.randn(1, 3, 224, 224)
outputs = model(inputs)

What is the likely mistake?

medium

A. Input tensor shape should be (3, 224, 224) without batch dimension.

B. You need to set weights=None to use pre-trained weights.

C. You forgot to call model.eval() before prediction.

D. You must convert inputs to numpy arrays before passing to model.

Solution

1. Step 1: Check model mode for prediction

   Pre-trained models must be set to evaluation mode with model.eval() to disable dropout and batch norm updates.

2. Step 2: Identify the missing step

   The code misses model.eval(), so outputs may be incorrect or inconsistent.

3. Final Answer:

   You forgot to call model.eval() before prediction. -> Option C

4. Quick Check:

   Set model.eval() before inference [OK]

Hint: Always call model.eval() before predicting [OK]

Common Mistakes:

• Not calling model.eval() before inference
• Wrong input tensor shape without batch
• Trying to convert tensors to numpy before model

5. You want to fine-tune a pre-trained ResNet18 model on your own 5-class dataset. Which of the following code snippets correctly replaces the final layer for this task?

hard

A. model.fc = torch.nn.Linear(in_features=512, out_features=5)

B. model.classifier = torch.nn.Linear(in_features=1000, out_features=5)

C. model.fc = torch.nn.Linear(in_features=2048, out_features=5)

D. model.output = torch.nn.Linear(in_features=512, out_features=1000)

Solution

1. Step 1: Identify the final layer of ResNet18

   ResNet18's final fully connected layer is model.fc with input features 512 and output 1000 classes.

2. Step 2: Replace final layer for 5 classes

   To fine-tune, replace model.fc with a new Linear layer with 512 inputs and 5 outputs.

3. Final Answer:

   model.fc = torch.nn.Linear(in_features=512, out_features=5) -> Option A

4. Quick Check:

   Replace model.fc with correct output size [OK]

Hint: Replace model.fc with Linear(512, number_of_classes) [OK]

Common Mistakes:

• Replacing wrong attribute like model.classifier
• Using wrong input feature size (2048 instead of 512)
• Not changing output features to dataset classes