Vision Transformer (ViT) in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Vision Transformer (ViT)

Which metric matters for Vision Transformer (ViT) and WHY

For Vision Transformers used in image classification, accuracy is the main metric. It tells us how many images the model labels correctly out of all images. However, when classes are uneven or some mistakes cost more, precision, recall, and F1 score become important. These metrics help us understand if the model is good at finding certain classes or avoiding wrong guesses.

Confusion Matrix Example

      | Predicted Cat | Predicted Dog |
      |--------------|---------------|
      | True Cat: 45 | False Dog: 5  |
      | False Cat: 3 | True Dog: 47  |

      Total samples = 45 + 5 + 3 + 47 = 100

      Precision (Cat) = TP / (TP + FP) = 45 / (45 + 3) = 0.9375
      Recall (Cat) = TP / (TP + FN) = 45 / (45 + 5) = 0.9
    

Precision vs Recall Tradeoff with Examples

Imagine ViT is used to detect rare animals in photos. If we want to be sure when the model says "animal found," we need high precision. This means fewer false alarms. But if missing any rare animal is bad, we want high recall to catch as many as possible, even if some guesses are wrong.

Choosing between precision and recall depends on the task. For example, in medical image analysis, missing a disease (low recall) is worse than a false alarm (low precision). For general object recognition, balanced metrics like F1 score help.

Good vs Bad Metric Values for ViT

Good: Accuracy above 85% on a balanced dataset means the ViT is learning well. Precision and recall above 80% show it finds and labels classes reliably.

Bad: Accuracy near random chance (e.g., 10% for 10 classes) means the model is not learning. Very high accuracy but low recall means it misses many true cases. Low precision means many wrong guesses.

Common Pitfalls in Metrics for ViT

• Accuracy paradox: High accuracy can hide poor performance if classes are imbalanced.
• Data leakage: If test images are too similar to training, metrics look better but model won't generalize.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training images, not learning patterns.

Self-Check Question

Your ViT model has 98% accuracy but only 12% recall on a rare class. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses most true cases of the rare class (low recall), which can be critical depending on the task. High accuracy is misleading because the rare class is small compared to others.

Key Result

Accuracy is key for ViT image classification, but precision and recall reveal deeper performance, especially on rare or important classes.