Custom object detection dataset in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Custom object detection dataset

Which metric matters for Custom Object Detection Dataset and WHY

For object detection, the key metric is mean Average Precision (mAP). It measures how well the model finds and correctly labels objects in images. mAP combines both precision (how many detected objects are correct) and recall (how many real objects are found). This balance is important because detecting too many wrong objects or missing real ones both hurt performance.

Confusion Matrix for Object Detection

Object detection uses a special confusion matrix based on True Positives (TP), False Positives (FP), and False Negatives (FN) per class. Here is a simple example for one class:

      +----------------+----------------+
      |                | Predicted Obj  |
      |                | Present | Absent |
      +----------------+---------+--------+
      | Actual Obj     |         |        |
      | Present       |   TP    |   FN   |
      | Absent        |   FP    |   TN*  |
      +----------------+---------+--------+

      *TN (True Negative) is less meaningful in object detection because background is large.
    

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

Precision vs Recall Tradeoff with Examples

In object detection:

• High Precision, Low Recall: The model finds fewer objects but is usually correct. Good when false alarms are costly, like detecting defects in products.
• High Recall, Low Precision: The model finds most objects but with many mistakes. Useful when missing any object is bad, like detecting pedestrians for safety.

Choosing the right balance depends on the task's needs.

What Good vs Bad Metric Values Look Like

For a custom object detection dataset:

• Good: mAP above 0.7 means the model detects most objects correctly with few mistakes.
• Bad: mAP below 0.3 means the model misses many objects or has many false detections.
• Precision and recall should both be reasonably high (above 0.6) for balanced performance.

Common Pitfalls in Metrics for Custom Object Detection

• Ignoring IoU Threshold: mAP depends on Intersection over Union (IoU) threshold. Too low threshold inflates scores; too high makes detection too strict.
• Data Leakage: Using test images in training leads to unrealistically high metrics.
• Class Imbalance: Rare classes may have low recall but get overshadowed in overall mAP.
• Overfitting: High training mAP but low test mAP means model memorizes training data, not generalizing.

Self Check

Your model has 98% accuracy but 12% recall on detecting a rare object. Is it good?

Answer: No. High accuracy can be misleading if most images don’t have the object. Low recall means the model misses most objects, which is bad for detection tasks.

Key Result

Mean Average Precision (mAP) is the key metric balancing precision and recall for custom object detection datasets.