Why segmentation labels every pixel in Computer Vision - Why Metrics Matter

Metrics & Evaluation - Why segmentation labels every pixel

Which metric matters for this concept and WHY

In image segmentation, every pixel is labeled to understand exactly what part of the image belongs to which object or background. The key metrics are Pixel Accuracy and Intersection over Union (IoU). Pixel Accuracy measures how many pixels are correctly labeled out of all pixels. IoU measures the overlap between the predicted and true regions for each class. These metrics matter because segmentation cares about detailed, pixel-level correctness, not just overall image labels.

Confusion matrix or equivalent visualization (ASCII)

       | Predicted Class A | Predicted Class B | Predicted Background
-------|-------------------|-------------------|---------------------
True A |        5000       |        300        |         200         
True B |        400        |       4500        |         100         
True BG|        100        |        150        |        4800         

Here, each cell counts pixels. For example, 5000 pixels truly are Class A and predicted correctly as Class A (True Positives for Class A). The matrix helps calculate precision, recall, and IoU for each class.

Precision vs Recall tradeoff with concrete examples

Imagine segmenting a road in a self-driving car image:

• High Precision: The model labels pixels as road only when very sure. Few false road pixels appear, but some road pixels might be missed (lower recall).
• High Recall: The model labels many pixels as road, catching almost all road pixels but also including some non-road pixels (lower precision).

For safety, high recall is often more important to avoid missing road areas, but too low precision can confuse the system. Balancing these is key.

What "good" vs "bad" metric values look like for this use case

Good segmentation: Pixel Accuracy above 90%, IoU above 70% for main classes. This means most pixels are correctly labeled and predicted regions overlap well with true regions.

Bad segmentation: Pixel Accuracy below 70%, IoU below 40%. Many pixels are mislabeled, and predicted regions poorly match true objects, making the segmentation unreliable.

Metrics pitfalls

• Accuracy paradox: If background pixels dominate, high pixel accuracy can be misleading because the model may just predict background well but fail on objects.
• Data leakage: Using test images seen during training inflates metrics falsely.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes training images and does not generalize.

Self-check question

Your segmentation model has 98% pixel accuracy but only 12% recall on the object class. Is it good for production? Why not?

Answer: No, it is not good. The high accuracy likely comes from correctly labeling background pixels, which are many. But the very low recall means the model misses most object pixels, failing to segment the object properly. This is a serious problem for tasks needing precise object detection.

Key Result

Pixel-level metrics like IoU and pixel accuracy are essential to evaluate segmentation quality because every pixel's label matters.