Dataset bias in vision in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Dataset bias in vision

Which metric matters for Dataset Bias in Vision and WHY

When dealing with dataset bias in vision tasks, accuracy alone can be misleading. Instead, precision, recall, and F1 score are important to understand how well the model performs across different groups or classes. This helps reveal if the model favors some categories over others due to bias in the data.

Also, confusion matrices help visualize errors per class, showing if some classes are systematically misclassified because of bias.

Confusion Matrix Example

      Actual \ Predicted | Cat | Dog | Rabbit
      -------------------------------------
      Cat                | 45  |  5  |  0
      Dog                | 10  | 30  | 10
      Rabbit             |  0  |  5  | 40
    

This matrix shows the model predicts cats well but confuses dogs and rabbits more. If the dataset had fewer rabbit images, the model might be biased against rabbits.

Precision vs Recall Tradeoff with Dataset Bias

Imagine a vision model detecting rare animals. If the dataset has few examples of rare animals, the model might have:

• High precision but low recall for rare animals: It only predicts rare animals when very sure, missing many actual rare animals.
• High recall but low precision: It predicts many rare animals, but many are wrong.

Dataset bias often causes low recall for underrepresented classes, meaning the model misses many true cases.

Good vs Bad Metric Values for Dataset Bias in Vision

Good: Balanced precision and recall across all classes, showing the model treats all categories fairly.

Bad: Very high accuracy but very low recall on minority classes, indicating the model ignores or misclassifies those classes due to bias.

Common Pitfalls in Metrics Due to Dataset Bias

• Accuracy paradox: High overall accuracy hides poor performance on minority classes.
• Data leakage: If biased features leak into training, metrics may look good but fail in real use.
• Overfitting to majority class: Model memorizes common classes, ignoring rare ones.

Self Check

Your vision model has 98% accuracy but only 12% recall on a rare animal class. Is it good for production?

Answer: No. The model misses most rare animals, which is critical if detecting them matters. High accuracy is misleading because the rare class is small but important.

Key Result

Balanced precision and recall across classes reveal dataset bias better than accuracy alone.