Point cloud processing in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Point cloud processing

Which metric matters for Point Cloud Processing and WHY

Point cloud processing often involves tasks like classification, segmentation, or object detection in 3D space. The key metrics depend on the task:

• For classification: Accuracy, Precision, Recall, and F1-score matter to understand how well the model identifies correct classes.
• For segmentation: Intersection over Union (IoU) or mean IoU is important to measure how well predicted 3D regions match the true regions.
• For detection: Precision and Recall are critical to balance false positives and false negatives in detecting objects.

These metrics help us know if the model correctly understands the 3D shapes and objects from point clouds.

Confusion Matrix Example for Point Cloud Classification

    Actual \ Predicted | Car | Pedestrian | Tree | Total
    -------------------|-----|------------|------|------
    Car                | 50  | 5          | 0    | 55
    Pedestrian         | 3   | 40         | 2    | 45
    Tree               | 0   | 1          | 49   | 50
    -------------------|-----|------------|------|------
    Total              | 53  | 46         | 51   | 150
    

From this matrix:

• True Positives (TP) for Car = 50
• False Positives (FP) for Car = 3 (Pedestrian predicted as Car) + 0 (Tree predicted as Car) = 3
• False Negatives (FN) for Car = 5 (Car predicted as Pedestrian) + 0 (Car predicted as Tree) = 5

Precision for Car = 50 / (50 + 3) ≈ 0.943

Recall for Car = 50 / (50 + 5) ≈ 0.909

Precision vs Recall Tradeoff in Point Cloud Tasks

Imagine a self-driving car using point cloud data to detect pedestrians:

• High Precision: The model rarely mistakes other objects for pedestrians. This avoids false alarms but might miss some real pedestrians.
• High Recall: The model detects almost all pedestrians, even if it sometimes mistakes other objects as pedestrians.

For safety, high recall is often more important to avoid missing any pedestrian, even if it means some false alarms.

What Good vs Bad Metrics Look Like for Point Cloud Processing

• Good: Accuracy > 90%, Precision and Recall both above 85%, IoU above 75% for segmentation tasks.
• Bad: Accuracy below 70%, Precision or Recall below 50%, IoU below 50%, indicating poor understanding of 3D shapes.

Good metrics mean the model reliably recognizes and segments objects in 3D space. Bad metrics mean it often confuses or misses objects.

Common Pitfalls in Metrics for Point Cloud Processing

• Accuracy Paradox: If one class dominates (like ground points), high accuracy can be misleading.
• Data Leakage: Using test points too similar to training points can inflate metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data, not generalizing well.
• Ignoring Class Imbalance: Some classes may have fewer points, so metrics like F1-score or IoU per class are better than overall accuracy.

Self Check: Your model has 98% accuracy but 12% recall on pedestrian detection. Is it good?

No, this is not good for pedestrian detection. The high accuracy likely comes from many non-pedestrian points correctly classified. But 12% recall means the model misses 88% of pedestrians, which is dangerous for safety-critical applications like self-driving cars.

Key Result

For point cloud tasks, precision, recall, and IoU are key to measure correct 3D object recognition and segmentation.