Why 3D understanding enables robotics and AR in Computer Vision - Why Metrics Matter

Metrics & Evaluation - Why 3D understanding enables robotics and AR

Which metric matters for this concept and WHY

For 3D understanding in robotics and AR, accuracy of depth estimation and object localization is key. This means how close the model's predicted 3D positions are to the real world. Good accuracy ensures robots can safely navigate and AR objects align well with the environment.

Additionally, precision and recall matter when detecting objects in 3D space. Precision ensures the system does not mistake background or noise for real objects, while recall ensures it finds all important objects to interact with.

Confusion matrix or equivalent visualization (ASCII)

    3D Object Detection Confusion Matrix:

                 Predicted
               | Object | No Object |
    Actual  -------------------------
    Object   |   TP   |    FN     |
    No Obj  |   FP   |    TN     |

    Example numbers:
    TP = 85 (correctly detected objects)
    FP = 10 (false alarms)
    FN = 5  (missed objects)
    TN = 100 (correctly ignored background)

    Total samples = 85 + 10 + 5 + 100 = 200
    

From this, precision = 85 / (85 + 10) = 0.895, recall = 85 / (85 + 5) = 0.944.

Precision vs Recall tradeoff with concrete examples

In robotics and AR, missing an object (low recall) can cause collisions or wrong interactions. So, high recall is important to find all objects.

But too many false alarms (low precision) can confuse the system, making it react to things that aren't there. This wastes resources and reduces user trust.

For example, a robot vacuum must detect furniture accurately. Missing a chair (low recall) causes bumping. Detecting a shadow as a chair (low precision) causes unnecessary stops.

Balancing precision and recall depends on the task. Navigation favors recall, while AR overlay quality favors precision.

What "good" vs "bad" metric values look like for this use case

• Good: Precision and recall above 90% means the system reliably detects and locates objects in 3D space.
• Bad: Precision below 70% means many false detections, confusing the robot or AR system.
• Bad: Recall below 70% means many objects are missed, risking collisions or poor AR alignment.
• Good: Depth estimation error under a few centimeters ensures accurate placement and navigation.
• Bad: Large depth errors cause AR objects to float or sink incorrectly and robots to misjudge distances.

Metrics pitfalls

• Accuracy paradox: In scenes with few objects, high accuracy can be misleading if the model just predicts "no object" everywhere.
• Data leakage: Training on scenes too similar to test scenes inflates metrics but fails in real-world diverse environments.
• Overfitting: Model performs well on training data but poorly on new scenes, showing low generalization.
• Ignoring spatial errors: Only counting detection misses depth or position errors that matter in 3D tasks.

Self-check question

Your 3D object detection model has 98% accuracy but only 12% recall on objects. Is it good for robotics or AR? Why or why not?

Answer: No, it is not good. The high accuracy likely comes from correctly identifying background (no object) most of the time. But the very low recall means it misses almost all objects, which is dangerous for robots and ruins AR experiences because objects are not detected or tracked properly.

Key Result

High recall and precision in 3D object detection are essential for safe and accurate robotics and AR applications.