Fine-tuning approach in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Fine-tuning approach

Which metric matters for Fine-tuning approach and WHY

When fine-tuning a computer vision model, the key metrics to watch are accuracy, precision, and recall. Accuracy shows how often the model predicts correctly overall. Precision tells us how many of the positive predictions are actually correct. Recall shows how many of the actual positive cases the model finds.

We focus on these because fine-tuning adjusts a pre-trained model to a new task or dataset. We want to see if the model improves in recognizing the new classes without losing its ability to avoid mistakes. Depending on the task, precision or recall might be more important. For example, in medical image diagnosis, recall is critical to catch all cases.

Confusion matrix example

    | Predicted Positive | Predicted Negative |
    |--------------------|--------------------|
    | True Positive (TP): 80  | False Negative (FN): 20 |
    | False Positive (FP): 10 | True Negative (TN): 90  |

    Total samples = TP + FP + TN + FN = 80 + 10 + 90 + 20 = 200

    Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
    Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
    Accuracy = (TP + TN) / Total = (80 + 90) / 200 = 0.85
    

Precision vs Recall tradeoff with examples

Fine-tuning can improve either precision or recall, but often improving one lowers the other. For example:

• High precision, low recall: The model is very sure when it says an object is present, but it misses some objects. Good for tasks where false alarms are costly, like detecting defects in manufacturing.
• High recall, low precision: The model finds almost all objects but also makes more mistakes. Useful in safety-critical tasks like detecting tumors, where missing any tumor is worse than a false alarm.

Fine-tuning helps balance this tradeoff by adjusting the model to the new data and task.

What "good" vs "bad" metric values look like for Fine-tuning

Good metrics:

• Accuracy above 85% on the new task dataset
• Precision and recall both above 80%, showing balanced performance
• F1 score (harmonic mean of precision and recall) above 0.8

Bad metrics:

• Accuracy below 70%, indicating poor adaptation
• Very low recall (e.g., below 50%), meaning many positives missed
• Very low precision (e.g., below 50%), meaning many false alarms
• Large gap between precision and recall, showing imbalance

Common pitfalls in Fine-tuning metrics

• Accuracy paradox: High accuracy can be misleading if the dataset is imbalanced. For example, if 95% of images are negative, a model always predicting negative gets 95% accuracy but is useless.
• Data leakage: If test images are too similar to training images, metrics look better but the model won't generalize.
• Overfitting: Fine-tuning too long on a small dataset can cause the model to memorize training images, showing high training accuracy but poor test accuracy.
• Ignoring class imbalance: Not using metrics like precision, recall, or F1 can hide poor performance on rare classes.

Self-check question

Your fine-tuned model has 98% accuracy but only 12% recall on the positive class (e.g., detecting defects). Is this good for production? Why or why not?

Answer: No, this is not good. The model misses 88% of the positive cases, which is very risky if detecting defects is important. High accuracy is misleading here because the dataset likely has many negatives. You need to improve recall to catch more positives.

Key Result

Fine-tuning success is best judged by balanced precision and recall, not just accuracy.

Practice

1. What is the main purpose of fine-tuning a pre-trained computer vision model?

easy

A. To adapt the model to a new task using less data and time

B. To train a model from scratch with a large dataset

C. To increase the size of the model for better accuracy

D. To remove layers from the model to make it smaller

Solution

1. Step 1: Understand fine-tuning concept

   Fine-tuning means starting from a model already trained on a related task.

2. Step 2: Identify the benefit

   This approach saves time and data by reusing learned features for a new task.

3. Final Answer:

   To adapt the model to a new task using less data and time -> Option A

4. Quick Check:

   Fine-tuning = adapt pre-trained model fast [OK]

Hint: Fine-tuning means reusing a model to learn new tasks faster [OK]

Common Mistakes:

• Thinking fine-tuning trains from scratch
• Assuming fine-tuning always increases model size
• Confusing fine-tuning with pruning layers

2. Which code snippet correctly freezes the layers of a PyTorch model before fine-tuning?

easy

A. for param in model.parameters(): param.requires_grad = False

B. model.freeze_layers()

C. model.trainable = False

D. for layer in model.layers: layer.trainable = True

Solution

1. Step 1: Recall PyTorch freezing syntax

   In PyTorch, freezing means setting requires_grad = False for parameters.

2. Step 2: Match code to syntax

   for param in model.parameters(): param.requires_grad = False correctly loops over parameters and disables gradient updates.

3. Final Answer:

   for param in model.parameters(): param.requires_grad = False -> Option A

4. Quick Check:

   Freeze layers = requires_grad False [OK]

Hint: Freeze layers by setting requires_grad = False in PyTorch [OK]

Common Mistakes:

• Using non-existent methods like freeze_layers()
• Setting model.trainable instead of parameters
• Confusing trainable True/False for freezing

3. Given this PyTorch code snippet for fine-tuning, what will be the output of print(sum(p.requires_grad for p in model.parameters())) after freezing layers?

medium

A. Raises an error

B. Number of all model parameters

C. 0

D. Number of unfrozen parameters

Solution

1. Step 1: Understand freezing effect on requires_grad

   Freezing sets requires_grad = False for all parameters.

2. Step 2: Calculate sum of requires_grad flags

   Since all are False, sum counts zero True values.

3. Final Answer:

   0 -> Option C

4. Quick Check:

   All frozen means requires_grad sum = 0 [OK]

Hint: Frozen layers have requires_grad = False, sum is zero [OK]

Common Mistakes:

• Assuming sum counts total parameters
• Thinking sum counts unfrozen parameters without freezing
• Expecting an error from requires_grad attribute

4. You tried fine-tuning but the model's accuracy did not improve. Which mistake could cause this?

medium

A. Using a pre-trained model instead of training from scratch

B. Freezing all layers and not unfreezing any

C. Adding more layers without training them

D. Using a very high learning rate during fine-tuning

Solution

1. Step 1: Identify learning rate impact

   A very high learning rate can cause unstable training and no improvement.

2. Step 2: Evaluate other options

   Freezing all layers prevents learning but usually keeps baseline accuracy; pre-trained models help; adding untrained layers alone doesn't prevent improvement if trained.

3. Final Answer:

   Using a very high learning rate during fine-tuning -> Option D

4. Quick Check:

   High learning rate = no improvement [OK]

Hint: Use smaller learning rates for fine-tuning to improve accuracy [OK]

Common Mistakes:

• Ignoring learning rate effects
• Assuming freezing all layers always improves
• Thinking training from scratch is better always

5. You want to fine-tune a pre-trained CNN for a new image classification task with 5 classes. Which sequence of steps is best practice?

hard

A. Train entire model from scratch with random weights for 5 classes

B. Freeze all layers, replace final layer with 5 outputs, train only final layer, then unfreeze some layers and fine-tune with low learning rate

C. Replace final layer with 5 outputs and train all layers at once with high learning rate

D. Freeze final layer, train earlier layers only, then unfreeze final layer

Solution

1. Step 1: Replace final layer for new classes

   Adjust output layer to match 5 classes for the new task.

2. Step 2: Freeze old layers and train new layer first

   This preserves learned features and trains new output layer quickly.

3. Step 3: Unfreeze some layers and fine-tune with low learning rate

   This improves model performance by adapting features carefully without large updates.

4. Final Answer:

   Freeze all layers, replace final layer with 5 outputs, train only final layer, then unfreeze some layers and fine-tune with low learning rate -> Option B

5. Quick Check:

   Stepwise fine-tuning with low LR = best practice [OK]

Hint: Freeze, replace output, train new layer, then unfreeze and fine-tune [OK]

Common Mistakes:

• Training all layers at once with high learning rate
• Training from scratch ignoring pre-trained weights
• Freezing final layer instead of earlier layers