Segmentation evaluation (IoU, Dice) in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Segmentation evaluation (IoU, Dice)

Which metric matters for segmentation and WHY

For segmentation tasks, we want to see how well the predicted area matches the true area. Two main metrics help us do this:

• IoU (Intersection over Union): It measures the overlap between the predicted and true segments divided by their combined area. It tells us how much the prediction and truth agree.
• Dice coefficient: It is similar but gives more weight to the overlap. It is twice the overlap divided by the total size of both areas.

Both metrics range from 0 to 1, where 1 means perfect match. They help us understand how accurate the segmentation is in a clear way.

Confusion matrix for segmentation (pixel-wise)

      |                | Predicted Positive | Predicted Negative |
      |----------------|--------------------|--------------------|
      | Actual Positive| True Positive (TP)  | False Negative (FN) |
      | Actual Negative| False Positive (FP) | True Negative (TN)  |

      IoU = TP / (TP + FP + FN)
      Dice = 2 * TP / (2 * TP + FP + FN)
    

Here, each pixel is counted as positive if it belongs to the object and negative if it does not. TP means pixels correctly predicted as object, FP means pixels wrongly predicted as object, and FN means pixels missed.

Tradeoff: IoU vs Dice with examples

Both IoU and Dice measure overlap but differ slightly:

• IoU is stricter. If the prediction is smaller or larger than the true area, IoU drops more.
• Dice is more forgiving and often higher than IoU for the same prediction.

Example: If your model predicts a tumor area in a scan:

• IoU tells you exactly how much the predicted tumor overlaps the real tumor.
• Dice gives a smoother score that can be easier to optimize during training.

Choosing between them depends on your task needs. Dice is popular in medical imaging, IoU in general object detection.

What good vs bad metric values look like

• Good: IoU or Dice above 0.8 means the predicted segment closely matches the true segment.
• Moderate: Scores between 0.5 and 0.8 show partial overlap but room for improvement.
• Bad: Scores below 0.5 mean poor overlap, predictions are often missing or wrong.

For example, a Dice score of 0.9 means 90% overlap, which is excellent for tasks like tumor segmentation.

Common pitfalls in segmentation metrics

• Ignoring class imbalance: If the object is very small, high Dice or IoU can be hard to achieve and accuracy can be misleading.
• Overfitting: Very high scores on training data but low on new images mean the model memorized rather than learned.
• Boundary errors: Small shifts in edges can reduce IoU a lot even if the shape looks visually good.
• Data leakage: Testing on images seen during training inflates metrics falsely.

Self-check question

Your model has an IoU of 0.95 on training images but only 0.4 on new images. Is it good for production? Why or why not?

Answer: No, this shows overfitting. The model performs well on known data but poorly on new data, so it will not generalize well in real use.

Key Result

IoU and Dice measure how well predicted segments overlap true segments; high values (close to 1) mean better segmentation quality.

Practice

1. What does the Intersection over Union (IoU) metric measure in image segmentation?

easy

A. The overlap between predicted and true masks divided by their union

B. The difference between predicted and true masks

C. The total number of pixels in the predicted mask

D. The ratio of true positives to false negatives

Solution

1. Step 1: Understand IoU definition

   IoU measures how much the predicted mask and true mask overlap compared to their combined area.

2. Step 2: Apply IoU formula

   IoU = (Area of Overlap) / (Area of Union) between predicted and true masks.

3. Final Answer:

   The overlap between predicted and true masks divided by their union -> Option A

4. Quick Check:

   IoU = Overlap / Union [OK]

Hint: IoU = Overlap divided by combined area [OK]

Common Mistakes:

• Confusing IoU with difference or subtraction
• Using only predicted mask area
• Mixing IoU with accuracy or precision

2. Which of the following is the correct formula for Dice coefficient between predicted mask P and true mask T?

easy

A. Dice = |P| / |T|

B. Dice = (2 * |P ∩ T|) / (|P| + |T|)

C. Dice = |P ∪ T| / (|P| + |T|)

D. Dice = |P ∩ T| / |P ∪ T|

Solution

1. Step 1: Recall Dice coefficient formula

   Dice coefficient is defined as twice the overlap divided by the sum of sizes of predicted and true masks.

2. Step 2: Match formula to options

   Dice = (2 * |P ∩ T|) / (|P| + |T|) matches Dice = (2 * intersection) / (sum of areas).

3. Final Answer:

   Dice = (2 * |P ∩ T|) / (|P| + |T|) -> Option B

4. Quick Check:

   Dice = 2*Overlap / (Sum areas) [OK]

Hint: Dice doubles overlap, divides by sum of mask sizes [OK]

Common Mistakes:

• Confusing Dice with IoU formula
• Using union instead of sum of areas
• Forgetting to multiply overlap by 2

3. Given predicted mask pixels: {1,2,3,4} and true mask pixels: {3,4,5,6}, what is the IoU score?

medium

A. 0.33

B. 0.25

C. 0.5

D. 0.4

Solution

1. Step 1: Calculate intersection and union

   Intersection = {3,4} size = 2; Union = {1,2,3,4,5,6} size = 6.

2. Step 2: Compute IoU

   IoU = Intersection / Union = 2 / 6 = 0.3333 (approx). Check options carefully.

3. Step 3: Re-check options

   0.33 is 0.33, closest to calculation. But 0.4 is 0.4, which is incorrect.

4. Final Answer:

   0.33 -> Option A

5. Quick Check:

   IoU = 2/6 = 0.33 [OK]

Hint: IoU = intersection size / union size [OK]

Common Mistakes:

• Counting union incorrectly
• Mixing intersection with union
• Rounding IoU incorrectly

4. You have this Python code to compute Dice score but it returns 0. Identify the error:

def dice_score(pred, true):
    intersection = len(pred & true)
    return intersection / (len(pred) + len(true))

pred = {1,2,3}
true = {2,3,4}
print(dice_score(pred, true))

medium

A. Wrong variable names in function

B. Using set intersection instead of union

C. Incorrect data type for masks

D. Missing multiplication by 2 in numerator

Solution

1. Step 1: Review Dice formula

   Dice = 2 * intersection / (sum of sizes). The code misses multiplying intersection by 2.

2. Step 2: Check code calculation

   Code returns intersection / sum, which is half the correct Dice score.

3. Final Answer:

   Missing multiplication by 2 in numerator -> Option D

4. Quick Check:

   Dice needs 2*intersection [OK]

Hint: Dice numerator must be 2 times intersection [OK]

Common Mistakes:

• Forgetting to multiply intersection by 2
• Confusing intersection with union
• Using wrong data types for masks

5. A segmentation model predicts a mask with 80 pixels, the true mask has 100 pixels, and their overlap is 60 pixels. What is the Dice score and IoU score respectively?

hard

A. Dice = 0.8, IoU = 0.7

B. Dice = 0.75, IoU = 0.6

C. Dice = 0.67, IoU = 0.5

D. Dice = 0.6, IoU = 0.75

Solution

1. Step 1: Calculate Dice score

   Dice = 2 * overlap / (predicted + true) = 2*60 / (80+100) = 120/180 = 0.6667 (approx). Check options carefully.

2. Step 2: Calculate IoU score

   IoU = overlap / union = 60 / (80 + 100 - 60) = 60 / 120 = 0.5.

3. Step 3: Re-check options

   Dice = 0.67, IoU = 0.5 matches Dice ~0.67 and IoU 0.5 exactly.

4. Final Answer:

   Dice = 0.67, IoU = 0.5 -> Option C

5. Quick Check:

   Dice = 2*60/180=0.67, IoU=60/120=0.5 [OK]

Hint: Dice doubles overlap; IoU divides by union [OK]

Common Mistakes:

• Using sum instead of union for IoU denominator
• Forgetting to multiply overlap by 2 for Dice
• Mixing Dice and IoU formulas