Non-maximum suppression in PyTorch - Deep Dive

Overview - Non-maximum suppression

What is it?

Non-maximum suppression (NMS) is a technique used to select the best bounding boxes from many overlapping boxes in object detection. It keeps the box with the highest confidence score and removes others that overlap too much with it. This helps reduce duplicate detections of the same object. NMS is essential for making object detection results clear and accurate.

Why it matters

Without NMS, object detection models would output many overlapping boxes for the same object, making it hard to understand what the model actually detected. This would confuse users and reduce the usefulness of detection systems in real-world tasks like self-driving cars or face recognition. NMS cleans up these results so the system can confidently say where objects are.

Where it fits

Before learning NMS, you should understand how object detection models predict bounding boxes and confidence scores. After NMS, learners often study more advanced post-processing techniques like soft-NMS or learn how to integrate NMS efficiently in model pipelines.

Mental Model

Core Idea

Non-maximum suppression picks the strongest detection and removes nearby weaker ones to avoid duplicates.

Think of it like...

Imagine you are picking the tallest person in a crowded room and asking everyone too close to them to step aside, so you only see one tall person clearly.

Detections: [■■■■■(score 0.9), ■■■■(0.8), ■■■(0.7)]
Overlap check → Keep highest score box ■■■■■
Remove boxes overlapping too much with ■■■■■
Result: Only ■■■■■ remains

Build-Up - 7 Steps

1

Foundation

What are bounding boxes and scores

Concept: Understanding the basic outputs of object detection models: boxes and confidence scores.

Object detection models predict rectangles (bounding boxes) around objects and assign a confidence score to each box. The score shows how sure the model is that the box contains an object.

Result

You get many boxes with scores, some overlapping the same object.

Knowing what bounding boxes and scores represent is essential before learning how to filter them.

2

Foundation

Why overlapping boxes cause problems

Concept: Recognizing that multiple boxes can cover the same object, causing confusion.

When a model detects an object, it often predicts several boxes around it with slightly different positions and scores. Without filtering, this looks like multiple detections for one object.

Result

Raw detection output has many overlapping boxes for the same object.

Understanding this problem motivates the need for a method like NMS.

3

Intermediate

How non-maximum suppression works

Concept: Learning the step-by-step process of NMS to select boxes.

NMS sorts boxes by score, picks the highest one, then removes all boxes that overlap too much with it (above a threshold). It repeats this until no boxes remain.

Result

A smaller set of boxes with minimal overlap, representing distinct objects.

Knowing the iterative filtering process clarifies how NMS cleans up detections.

4

Intermediate

Intersection over Union (IoU) explained

Concept: Understanding the overlap measure used to decide which boxes to remove.

IoU measures how much two boxes overlap by dividing the area of their intersection by the area of their union. A higher IoU means more overlap.

Result

You can quantify overlap to decide if a box should be suppressed.

Grasping IoU is key to controlling NMS behavior and tuning thresholds.

5

Intermediate

Implementing NMS in PyTorch

🤔Before reading on: do you think PyTorch has a built-in NMS function or do you need to write it from scratch? Commit to your answer.

Concept: Using PyTorch's built-in function to apply NMS efficiently.

PyTorch provides torchvision.ops.nms(boxes, scores, iou_threshold) which returns indices of boxes to keep. You pass boxes as tensors, scores, and an IoU threshold.

Result

You get filtered boxes ready for final detection output.

Knowing the built-in function saves time and ensures efficient, tested NMS.

6

Advanced

Choosing the IoU threshold wisely

🤔Before reading on: do you think a very low IoU threshold keeps more or fewer boxes? Commit to your answer.

Concept: Understanding how the IoU threshold affects detection results.

A low threshold removes boxes with even small overlap, possibly losing true positives. A high threshold keeps more boxes, risking duplicates. Balancing this threshold is crucial.

Result

Detection quality changes: too low causes missed objects, too high causes duplicates.

Knowing threshold impact helps tune NMS for best accuracy in different tasks.

7

Expert

Limitations and alternatives to standard NMS

🤔Before reading on: do you think standard NMS can handle overlapping objects well? Commit to your answer.

Concept: Exploring cases where NMS fails and advanced methods like soft-NMS or learned NMS.

Standard NMS can wrongly remove boxes when objects are close or overlapping. Soft-NMS reduces scores instead of removing boxes, keeping more detections. Learned NMS uses models to decide suppression.

Result

Better detection in crowded scenes and improved accuracy.

Understanding NMS limits guides choosing or designing better post-processing for complex scenarios.

Under the Hood

NMS works by sorting detection boxes by confidence scores, then iteratively selecting the highest scoring box and removing all boxes with IoU above a threshold with it. This process repeats until no boxes remain. Internally, this involves tensor operations for sorting, IoU calculation, and masking to filter boxes efficiently.

Why designed this way?

NMS was designed to solve the problem of multiple overlapping detections in a simple, fast way. Alternatives like clustering or learned suppression were more complex or slower. NMS balances speed and effectiveness, making it suitable for real-time systems.

┌───────────────┐
│ Input Boxes   │
│ + Scores     │
└──────┬────────┘
       │ Sort by score
       ▼
┌───────────────┐
│ Pick highest  │
│ scoring box   │
└──────┬────────┘
       │ Calculate IoU
       ▼
┌───────────────┐
│ Remove boxes  │
│ with IoU > T │
└──────┬────────┘
       │ Repeat until no boxes
       ▼
┌───────────────┐
│ Output boxes  │
│ after NMS     │
└───────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Does NMS always keep the box with the largest area? Commit to yes or no before reading on.

Common Belief: NMS keeps the biggest box because it covers the object best.

Reality: NMS keeps the box with the highest confidence score, not necessarily the biggest box.

Why it matters: Choosing boxes by size instead of score can keep less accurate detections, reducing model precision.

Quick: Does NMS remove all overlapping boxes regardless of their scores? Commit to yes or no before reading on.

Common Belief: NMS removes every box that overlaps with any other box.

Reality: NMS only removes boxes that overlap too much with a higher scoring box, not all overlapping boxes.

Why it matters: Removing all overlapping boxes would discard valid detections, hurting recall.

Quick: Can NMS perfectly separate objects that are very close together? Commit to yes or no before reading on.

Common Belief: NMS can always distinguish closely packed objects perfectly.

Reality: Standard NMS struggles with close or overlapping objects and may remove true positives.

Why it matters: This limitation can cause missed detections in crowded scenes, requiring advanced methods.

Expert Zone

1

NMS performance depends heavily on the IoU threshold, which often requires task-specific tuning.

2

The order of boxes after sorting affects which boxes are kept, so score calibration impacts results.

3

Batch processing NMS efficiently requires careful tensor operations to avoid slow loops.

When NOT to use

Standard NMS is not ideal when objects are densely packed or heavily overlapping. Alternatives like soft-NMS, which reduces scores instead of removing boxes, or learned NMS methods that use neural networks to decide suppression, are better choices.

Production Patterns

In production, NMS is often integrated as a final step in detection pipelines using optimized libraries like torchvision.ops.nms. Systems tune IoU thresholds per class and may combine NMS with confidence thresholding and class-wise filtering for best results.

Connections

Clustering algorithms

Both group similar items and reduce redundancy.

Understanding clustering helps grasp how NMS groups overlapping boxes and selects representatives.

Signal processing peak detection

NMS is similar to picking peaks in noisy signals by suppressing nearby lower peaks.

Knowing peak detection clarifies why NMS picks the strongest box and suppresses neighbors.

Human visual attention

NMS mimics how humans focus on the most prominent object and ignore close distractions.

This connection shows how AI mimics natural filtering to simplify complex scenes.

Common Pitfalls

#1 Using a very low IoU threshold causing missed detections.

Wrong approach:

indices = torchvision.ops.nms(boxes, scores, iou_threshold=0.1)

Correct approach:

indices = torchvision.ops.nms(boxes, scores, iou_threshold=0.5)

Root cause:

Misunderstanding that too low a threshold removes boxes that are actually distinct objects.

#2 Applying NMS before sorting boxes by score.

Wrong approach:

indices = torchvision.ops.nms(boxes, scores, iou_threshold=0.5) # boxes unsorted

Correct approach:

scores, order = scores.sort(descending=True)
boxes = boxes[order]
indices = torchvision.ops.nms(boxes, scores, iou_threshold=0.5)

Root cause:

Not sorting boxes causes NMS to keep wrong boxes because it assumes sorted input.

#3 Ignoring class labels and applying NMS across all classes together.

Wrong approach:

indices = torchvision.ops.nms(boxes, scores, iou_threshold=0.5) # mixed classes

Correct approach:

for cls in unique_classes:
cls_mask = labels == cls
cls_indices = torchvision.ops.nms(boxes[cls_mask], scores[cls_mask], iou_threshold=0.5)
# combine cls_indices

Root cause:

Applying NMS across classes removes valid detections from different object types.

Key Takeaways

Non-maximum suppression cleans up overlapping detection boxes by keeping the highest scoring ones and removing others that overlap too much.

IoU is the key measure to decide how much overlap is too much, and tuning its threshold affects detection quality.

PyTorch provides a built-in efficient NMS function that should be used instead of custom implementations.

Standard NMS struggles with crowded scenes, so alternatives like soft-NMS or learned NMS can improve results.

Applying NMS correctly requires sorting boxes by score and handling classes separately to avoid removing valid detections.

Practice

1. What is the main purpose of Non-maximum Suppression (NMS) in object detection using PyTorch?

easy

A. To increase the number of detected boxes

B. To resize images before detection

C. To randomly select boxes for training

D. To remove overlapping bounding boxes and keep only the best ones

Solution

1. Step 1: Understand the role of NMS

   Non-maximum suppression is used to filter out overlapping bounding boxes that represent the same object.

2. Step 2: Identify the goal of NMS

   It keeps only the bounding box with the highest confidence score among overlapping boxes to avoid duplicate detections.

3. Final Answer:

   To remove overlapping bounding boxes and keep only the best ones -> Option D

4. Quick Check:

   NMS purpose = remove overlaps, keep best [OK]

Hint: NMS filters duplicates, keeping only top scoring boxes [OK]

Common Mistakes:

• Thinking NMS increases detected boxes
• Confusing NMS with image resizing
• Assuming NMS randomly selects boxes

2. Which PyTorch function is commonly used to perform Non-maximum Suppression on bounding boxes and scores?

easy

A. torch.sort

B. torch.max

C. torch.ops.torchvision.nms

D. torch.argmax

Solution

1. Step 1: Identify the PyTorch NMS function

   PyTorch provides a dedicated function called torch.ops.torchvision.nms to perform Non-maximum Suppression.

2. Step 2: Differentiate from other functions

   torch.max, torch.sort, and torch.argmax do not perform NMS; they serve different purposes like finding max values or sorting tensors.

3. Final Answer:

   torch.ops.torchvision.nms -> Option C

4. Quick Check:

   PyTorch NMS function = torch.ops.torchvision.nms [OK]

Hint: Use torch.ops.torchvision.nms for Non-maximum Suppression in PyTorch [OK]

Common Mistakes:

• Using torch.max instead of torch.ops.torchvision.nms
• Confusing sorting functions with NMS
• Trying torch.argmax for NMS

3. Given the following PyTorch code snippet, what will be the output tensor of indices after applying NMS?

import torch
boxes = torch.tensor([[10, 10, 20, 20], [11, 11, 21, 21], [30, 30, 40, 40]], dtype=torch.float)
scores = torch.tensor([0.9, 0.8, 0.7])
iou_threshold = 0.5
indices = torch.ops.torchvision.nms(boxes, scores, iou_threshold)
print(indices.tolist())

medium

A. [0, 2]

B. [1, 2]

C. [0, 1, 2]

D. [2]

Solution

1. Step 1: Analyze box overlaps and scores

   Boxes 0 and 1 overlap with IoU > 0.5, box 2 is separate. Box 0 has higher score (0.9) than box 1 (0.8).

2. Step 2: Apply NMS logic

   NMS keeps box 0, removes box 1 due to overlap and lower score, and keeps box 2 as it does not overlap.

3. Final Answer:

   [0, 2] -> Option A

4. Quick Check:

   Keep highest score boxes without high overlap = [0, 2] [OK]

Hint: Keep highest score boxes, remove overlapping lower scores [OK]

Common Mistakes:

• Keeping all boxes ignoring overlap
• Removing box 0 instead of box 1
• Keeping only one box

4. Identify the error in this PyTorch NMS usage and select the correct fix:

import torch
boxes = torch.tensor([[0, 0, 10, 10], [1, 1, 11, 11]], dtype=torch.float)
scores = torch.tensor([0.5, 0.6])
iou_threshold = 0.3
indices = torch.nms(boxes, scores, iou_threshold)
print(indices)

medium

A. Set iou_threshold to 1.0

B. Use torch.ops.torchvision.nms instead of torch.nms

C. Change boxes dtype to int

D. Sort scores before calling nms

Solution

1. Step 1: Check function availability

   torch.nms is not a direct PyTorch function; NMS is accessed via torchvision ops as torch.ops.torchvision.nms.

2. Step 2: Correct function call

   Replacing torch.nms with torch.ops.torchvision.nms fixes the error and runs NMS correctly.

3. Final Answer:

   Use torch.ops.torchvision.nms instead of torch.nms -> Option B

4. Quick Check:

   Correct NMS call = torch.ops.torchvision.nms [OK]

Hint: Use torchvision's nms via torch.ops.torchvision.nms [OK]

Common Mistakes:

• Calling torch.nms directly
• Changing box dtype unnecessarily
• Adjusting IoU threshold without fixing function

5. You have detected multiple overlapping boxes with scores: [0.95, 0.9, 0.85, 0.6]. If you apply Non-maximum Suppression with a very low IoU threshold (e.g., 0.1), what is the expected effect on the number of boxes kept?

hard

A. Most boxes will be removed because even slight overlaps are suppressed

B. All boxes will be kept because the threshold is low

C. Only the box with the lowest score will be kept

D. Boxes will be randomly removed regardless of overlap

Solution

1. Step 1: Understand IoU threshold meaning

   A low IoU threshold means boxes with even small overlaps are considered duplicates and suppressed.

2. Step 2: Predict effect on boxes

   Because boxes overlap slightly, most will be removed except the highest scoring box, reducing the number kept.

3. Final Answer:

   Most boxes will be removed because even slight overlaps are suppressed -> Option A

4. Quick Check:

   Low IoU threshold = strict filtering = fewer boxes [OK]

Hint: Lower IoU threshold means stricter overlap removal [OK]

Common Mistakes:

• Thinking low threshold keeps more boxes
• Assuming random removal
• Believing lowest score box is kept