Depth estimation basics in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Depth estimation basics

Which metric matters for depth estimation and WHY

Depth estimation predicts how far objects are in an image. We want to know how close the predicted depths are to the true depths.

Common metrics include:

• Mean Absolute Error (MAE): Average of absolute differences between predicted and true depths. Lower is better.
• Root Mean Squared Error (RMSE): Square root of average squared differences. Penalizes big mistakes more.
• Threshold Accuracy: Percentage of pixels where prediction is within a certain ratio of true depth (e.g., within 1.25 times). Higher is better.

These metrics tell us how accurate and reliable the depth predictions are.

Confusion matrix or equivalent visualization

Depth estimation is a regression task, so confusion matrix is not used. Instead, we visualize errors like this:

True Depth:      [2.0, 3.5, 1.0, 4.0]
Predicted Depth: [2.1, 3.0, 1.2, 5.0]

Errors (abs):    [0.1, 0.5, 0.2, 1.0]
MAE = (0.1+0.5+0.2+1.0)/4 = 0.45
RMSE = sqrt((0.1**2 + 0.5**2 + 0.2**2 + 1.0**2)/4) ≈ 0.57

Threshold accuracy @1.25:
Check if max(pred/true, true/pred) < 1.25
Values: [1.05, 1.17, 1.20, 1.25]
3 out of 4 pass -> 75% accuracy
    

Precision vs Recall tradeoff (or equivalent) with concrete examples

Depth estimation does not use precision and recall because it is not classification. Instead, we balance:

• Small average error (MAE/RMSE): Means predictions are close on average.
• High threshold accuracy: Means most predictions are close enough within a tolerance.

For example, a robot navigating a room needs depth predictions that are mostly accurate (high threshold accuracy) to avoid obstacles safely.

If the model has low average error but many big mistakes, it might be risky. If it has high threshold accuracy but slightly higher average error, it might be safer.

What "good" vs "bad" metric values look like for depth estimation

Good values:

• MAE less than 0.1 meters (small average error)
• RMSE less than 0.15 meters (few large errors)
• Threshold accuracy @1.25 above 90% (most predictions close)

Bad values:

• MAE above 0.5 meters (large average error)
• RMSE above 0.7 meters (many big mistakes)
• Threshold accuracy @1.25 below 60% (many predictions far off)

Good metrics mean the model can reliably tell how far things are. Bad metrics mean the model is often wrong and not useful.

Common pitfalls in depth estimation metrics

• Ignoring scale differences: Depth predictions might be correct up to a scale factor but metrics expect absolute values.
• Using only average error: Can hide big mistakes if many predictions are close but some are very wrong.
• Data leakage: Testing on images very similar to training can give overly optimistic metrics.
• Overfitting: Model performs well on training data but poorly on new scenes, metrics look good but model is not general.

Self-check question

Your depth estimation model has 98% threshold accuracy @1.25 but an RMSE of 1.5 meters. Is it good for real-world use?

Answer: No. The high threshold accuracy means most predictions are close, but the large RMSE shows some predictions have very big errors. These big mistakes can cause problems in applications like robot navigation. The model needs improvement to reduce large errors.

Key Result

Depth estimation quality is best judged by low average errors (MAE, RMSE) and high threshold accuracy, balancing overall closeness and avoiding large mistakes.