LiDAR data processing basics in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - LiDAR data processing basics

Which metric matters for LiDAR data processing and WHY

In LiDAR data processing, common tasks include object detection, segmentation, and classification. The key metrics depend on the task:

• For classification: Precision and Recall matter to balance false alarms and missed objects.
• For segmentation: Intersection over Union (IoU) is important to measure how well predicted shapes match true shapes.
• For detection: Average Precision (AP) summarizes precision-recall tradeoff across thresholds.

These metrics help us understand if the model correctly identifies objects and their shapes from 3D point clouds, which is crucial for safe and accurate applications like self-driving cars.

Confusion matrix example for LiDAR object classification

    Actual \ Predicted | Car | Pedestrian | Background
    -----------------------------------------------
    Car                | 50  | 5          | 10
    Pedestrian         | 3   | 40         | 7
    Background         | 8   | 4          | 200
    

From this matrix:

• True Positives (TP) for Car = 50
• False Positives (FP) for Car = 3 + 8 = 11 (Pedestrian and Background predicted as Car)
• False Negatives (FN) for Car = 5 + 10 = 15 (Car predicted as Pedestrian or Background)

Precision for Car = TP / (TP + FP) = 50 / (50 + 11) ≈ 0.82

Recall for Car = TP / (TP + FN) = 50 / (50 + 15) ≈ 0.77

Precision vs Recall tradeoff in LiDAR data tasks

Imagine a self-driving car detecting pedestrians:

• High Precision: Few false alarms. The car rarely thinks something is a pedestrian when it is not. This avoids unnecessary stops.
• High Recall: Few missed detections. The car catches almost every pedestrian, which is safer.

But improving one often lowers the other. If the model is too cautious, it may miss pedestrians (low recall). If it is too sensitive, it may stop for harmless objects (low precision).

Choosing the right balance depends on the application's safety needs.

What good vs bad metric values look like for LiDAR classification

• Good: Precision and Recall above 0.85 means the model correctly finds most objects and rarely mistakes background for objects.
• Bad: Precision or Recall below 0.5 means many false alarms or many missed objects, which can be dangerous in real-world use.
• IoU: Values above 0.7 show good overlap between predicted and true object shapes; below 0.4 means poor segmentation.

Common pitfalls in LiDAR metrics

• Accuracy paradox: High accuracy can be misleading if most points are background. The model may predict background well but fail on objects.
• Data leakage: Using the same scenes in training and testing inflates metrics falsely.
• Overfitting: Very high training metrics but low test metrics show the model memorizes data instead of learning general patterns.
• Ignoring class imbalance: Many more background points than objects can bias metrics. Use balanced metrics like F1-score.

Self-check question

Your LiDAR object detection model has 98% accuracy but only 12% recall on pedestrians. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is likely because most points are background and predicted correctly. But 12% recall means the model misses 88% of pedestrians, which is unsafe for real-world use.

Key Result

Precision and recall are key to evaluate LiDAR models, ensuring objects are detected accurately without many misses or false alarms.

Practice

1. What does LiDAR data primarily represent in computer vision?

easy

A. A sequence of text commands for robots

B. A collection of 3D points showing object shapes and distances

C. A 2D image with color information

D. A sound wave pattern for audio analysis

Solution

1. Step 1: Understand LiDAR data basics

   LiDAR uses lasers to measure distances and creates 3D points representing shapes and distances.

2. Step 2: Compare options to definition

   Only A collection of 3D points showing object shapes and distances describes 3D points showing shapes and distances, matching LiDAR data.

3. Final Answer:

   A collection of 3D points showing object shapes and distances -> Option B

4. Quick Check:

   LiDAR = 3D points [OK]

Hint: LiDAR = 3D points, not images or sounds [OK]

Common Mistakes:

• Confusing LiDAR with 2D images
• Thinking LiDAR is audio data
• Assuming LiDAR is text commands

2. Which Python code snippet correctly filters LiDAR points with height above 2 meters from a list points where each point is (x, y, z)?

easy

A. filtered = [p for p in points if p[2] > 2]

B. filtered = [p for p in points if p[1] > 2]

C. filtered = [p for p in points if p[0] > 2]

D. filtered = [p for p in points if p[3] > 2]

Solution

1. Step 1: Identify height coordinate in point tuple

   Points are (x, y, z), where z is height, so index 2 is height.

2. Step 2: Check filtering condition

   Filter points where z > 2 means p[2] > 2, matching filtered = [p for p in points if p[2] > 2].

3. Final Answer:

   filtered = [p for p in points if p[2] > 2] -> Option A

4. Quick Check:

   Height is z = p[2] [OK]

Hint: Height is the third value in (x,y,z) tuples [OK]

Common Mistakes:

• Using wrong index for height
• Trying to access p[3] which is out of range
• Filtering by x or y instead of z

3. Given this Python code to calculate mean height from LiDAR points, what is the output?

points = [(1,2,3), (4,5,6), (7,8,9)]
mean_height = sum(p[2] for p in points) / len(points)
print(round(mean_height, 2))

medium

A. 8.0

B. 7.0

C. 5.0

D. 6.0

Solution

1. Step 1: Extract z values from points

   Heights are 3, 6, and 9 from each tuple's third element.

2. Step 2: Calculate mean height

   Sum = 3 + 6 + 9 = 18; count = 3; mean = 18 / 3 = 6.0

3. Final Answer:

   6.0 -> Option D

4. Quick Check:

   Mean height = 6.0 [OK]

Hint: Sum heights then divide by count for mean [OK]

Common Mistakes:

• Summing wrong coordinate index
• Dividing by wrong length
• Not rounding output

4. Find the error in this code that tries to filter LiDAR points below 1 meter height:

points = [(0,0,0.5), (1,1,1.5), (2,2,0.8)]
filtered = [p for p in points if p[3] < 1]
print(filtered)

medium

A. SyntaxError due to wrong list comprehension

B. No error, code works correctly

C. IndexError because p[3] does not exist

D. filtered list will be empty due to condition

Solution

1. Step 1: Check point tuple length

   Each point has 3 elements indexed 0,1,2; p[3] is out of range.

2. Step 2: Identify error type

   Accessing p[3] causes IndexError at runtime.

3. Final Answer:

   IndexError because p[3] does not exist -> Option C

4. Quick Check:

   Tuple length = 3, max index = 2 [OK]

Hint: Check tuple length before indexing [OK]

Common Mistakes:

• Assuming p[3] exists
• Thinking it's a syntax error
• Ignoring runtime errors

5. You want to remove noisy LiDAR points that are too far from the ground (height > 10 meters) and then calculate the average height of remaining points. Which sequence of steps is correct?

hard

A. Filter points with height ≤ 10, then compute mean height from filtered points

B. Compute mean height first, then filter points with height ≤ 10

C. Filter points with height > 10, then compute mean height from filtered points

D. Compute mean height of all points, ignoring filtering

Solution

1. Step 1: Filter noisy points correctly

   Remove points with height > 10 means keep points with height ≤ 10.

2. Step 2: Calculate mean height after filtering

   Compute average height only from filtered points to get accurate mean.

3. Final Answer:

   Filter points with height ≤ 10, then compute mean height from filtered points -> Option A

4. Quick Check:

   Filter first, then average [OK]

Hint: Filter noise before averaging [OK]

Common Mistakes:

• Averaging before filtering noise
• Filtering wrong height range
• Ignoring filtering step