Bounding box handling in PyTorch - Model Pipeline Trace

Model Pipeline - Bounding box handling

This pipeline shows how bounding boxes are processed in an object detection task. It starts with raw image data and bounding box coordinates, then normalizes and encodes the boxes, trains a model to predict boxes, and finally decodes predictions to get final bounding boxes.

Data Flow - 6 Stages

1Input data

1000 images x 3 channels x 224 height x 224 width→Load images and raw bounding box coordinates (xmin, ymin, xmax, ymax) per image→1000 images x 3 x 224 x 224 and 1000 sets of bounding boxes (variable count)

Image tensor shape: (3, 224, 224), bounding box example: [50, 30, 150, 180]

↓

2Preprocessing

1000 images x 3 x 224 x 224 and bounding boxes→Normalize images and convert bounding boxes to relative coordinates (0 to 1 scale)→1000 images x 3 x 224 x 224 and bounding boxes normalized to [0,1]

Bounding box [50, 30, 150, 180] becomes [0.22, 0.13, 0.68, 0.80]

↓

3Feature extraction

1000 images x 3 x 224 x 224→Pass images through CNN backbone to extract features→1000 images x 512 feature maps (flattened)

Feature vector example: [0.12, -0.05, 0.33, ...]

↓

4Bounding box encoding

Normalized bounding boxes→Encode bounding boxes relative to anchor boxes for training→Encoded bounding box targets matching model output shape

Encoded box: [0.1, -0.05, 0.2, 0.15]

↓

5Model training

Feature vectors and encoded bounding boxes→Train model to predict bounding box offsets and class scores→Model weights updated, predictions shape: batch x anchors x 4 (box offsets)

Predicted offsets: [0.09, -0.04, 0.18, 0.14]

↓

6Bounding box decoding

Predicted bounding box offsets→Decode offsets back to absolute bounding box coordinates→Predicted bounding boxes in image coordinate scale

Decoded box: [48, 28, 148, 178]

Training Trace - Epoch by Epoch

Loss
1.2 |****
0.9 |*** 
0.7 |**  
0.5 |*   
0.4 |    
     1  2  3  4  5  Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.45	Initial training with high loss and low accuracy
2	0.9	0.60	Loss decreased, accuracy improved
3	0.7	0.72	Model learning bounding box predictions better
4	0.5	0.80	Good convergence, bounding box localization improving
5	0.4	0.85	Training stabilizing with low loss and high accuracy

Prediction Trace - 4 Layers

Layer 1: Input image

Layer 2: Feature extraction CNN

Layer 3: Bounding box prediction layer

Layer 4: Bounding box decoding

Model Quiz - 3 Questions

What is the purpose of bounding box encoding during training?

A To extract features from images

B To normalize images to 0-1 range

C To convert bounding boxes into offsets relative to anchors

D To decode predicted boxes back to coordinates

Key Insight

Bounding box handling involves converting boxes to a form the model can predict (encoding), training the model to predict offsets, and then converting predictions back to usable coordinates (decoding). This process helps the model learn to locate objects accurately in images.

Practice

1. What do the four numbers in a bounding box usually represent in PyTorch object detection tasks?

easy

A. Width, height, depth, and color

B. Center x, center y, width, height

C. Top-left corner x and y, width, height

D. Left, top, right, bottom coordinates of the box

Solution

1. Step 1: Understand bounding box format

   Bounding boxes in PyTorch detection models are often represented by four numbers indicating the edges: left (x_min), top (y_min), right (x_max), and bottom (y_max).

2. Step 2: Compare options with standard format

   Left, top, right, bottom coordinates of the box matches this format exactly, while others describe different formats or unrelated values.

3. Final Answer:

   Left, top, right, bottom coordinates of the box -> Option D

4. Quick Check:

   Bounding box = left, top, right, bottom [OK]

Hint: Remember bounding box edges are left, top, right, bottom [OK]

Common Mistakes:

• Confusing center coordinates with edges
• Mixing width and height with coordinates
• Assuming depth or color are part of bounding box

2. Which PyTorch tensor shape correctly represents a batch of 5 bounding boxes, each with 4 coordinates?

easy

A. torch.Size([5, 4])

B. torch.Size([4, 5])

C. torch.Size([5])

D. torch.Size([4])

Solution

1. Step 1: Identify bounding box tensor shape

   Each bounding box has 4 coordinates. For 5 boxes, the tensor shape should be (5, 4), meaning 5 rows (boxes) and 4 columns (coordinates).

2. Step 2: Match options with correct shape

   torch.Size([5, 4]) matches (5, 4). torch.Size([4, 5]) reverses dimensions, and C and D are one-dimensional, which is incorrect.

3. Final Answer:

   torch.Size([5, 4]) -> Option A

4. Quick Check:

   Batch size 5 and 4 coords = (5, 4) [OK]

Hint: Batch size first, coordinates second in tensor shape [OK]

Common Mistakes:

• Swapping batch and coordinate dimensions
• Using 1D tensor for multiple boxes
• Confusing number of boxes with number of coordinates

3. Given the following PyTorch code, what is the output?

import torch
boxes = torch.tensor([[1, 2, 4, 6], [0, 0, 3, 3]])
widths = boxes[:, 2] - boxes[:, 0]
heights = boxes[:, 3] - boxes[:, 1]
areas = widths * heights
print(areas.tolist())

medium

A. [10, 9]

B. [15, 9]

C. [12, 9]

D. [12, 6]

Solution

1. Step 1: Calculate widths and heights

   For box 1: width = 4 - 1 = 3, height = 6 - 2 = 4. For box 2: width = 3 - 0 = 3, height = 3 - 0 = 3.

2. Step 2: Calculate areas

   Area box 1 = 3 * 4 = 12, area box 2 = 3 * 3 = 9.

3. Final Answer:

   [12, 9] -> Option C

4. Quick Check:

   Area = width * height = [12, 9] [OK]

Hint: Area = (right-left) * (bottom-top) for each box [OK]

Common Mistakes:

• Subtracting coordinates in wrong order
• Mixing width with height values
• Forgetting to multiply widths and heights

4. Identify the error in this PyTorch code snippet for checking if points lie inside bounding boxes:

import torch
boxes = torch.tensor([1, 1, 5, 5])
points = torch.tensor([[3, 3], [6, 6]])
inside = (points[:, 0] >= boxes[:, 0]) & (points[:, 0] <= boxes[:, 2]) & (points[:, 1] >= boxes[:, 1]) & (points[:, 1] <= boxes[:, 3])
print(inside)

medium

A. The comparison between points and boxes tensors will fail due to shape mismatch

B. The logical AND operator '&' cannot be used with tensors

C. The code should use .all() to combine conditions

D. Points coordinates are reversed in the comparisons

Solution

1. Step 1: Check tensor shapes for broadcasting

   Boxes shape is (4,), points shape is (2,2). Comparing points[:, 0] (shape 2) with boxes[:, 0] causes shape mismatch as boxes is 1D.

2. Step 2: Understand broadcasting rules

   Direct element-wise comparison fails because points and boxes dimensions don't align for broadcasting in this code.

3. Final Answer:

   The comparison between points and boxes tensors will fail due to shape mismatch -> Option A

4. Quick Check:

   Shape mismatch causes error in element-wise comparison [OK]

Hint: Check tensor shapes before element-wise comparisons [OK]

Common Mistakes:

• Assuming PyTorch auto-broadcasts all shapes
• Using bitwise operators incorrectly
• Mixing coordinate order in comparisons

5. You want to convert bounding boxes from (left, top, right, bottom) format to (center_x, center_y, width, height) in PyTorch. Which code snippet correctly performs this conversion for a tensor boxes of shape (N, 4)?

hard

A. centers = boxes[:, :2] - boxes[:, 2:]
sizes = boxes[:, 2:] + boxes[:, :2]
converted = torch.cat([centers, sizes], dim=1)

B. centers = (boxes[:, :2] + boxes[:, 2:]) / 2
sizes = boxes[:, 2:] - boxes[:, :2]
converted = torch.cat([centers, sizes], dim=1)

C. centers = boxes[:, 2:] - boxes[:, :2]
sizes = (boxes[:, :2] + boxes[:, 2:]) / 2
converted = torch.cat([sizes, centers], dim=1)

D. centers = boxes[:, :2] * boxes[:, 2:]
sizes = boxes[:, 2:] / boxes[:, :2]
converted = torch.cat([centers, sizes], dim=1)

Solution

1. Step 1: Calculate center coordinates

   Center x and y are the average of left and right, top and bottom: (left + right)/2, (top + bottom)/2, which is (boxes[:, :2] + boxes[:, 2:]) / 2.

2. Step 2: Calculate width and height

   Width and height are right - left and bottom - top: boxes[:, 2:] - boxes[:, :2].

3. Step 3: Concatenate centers and sizes

   Combine centers and sizes along dimension 1 to get (center_x, center_y, width, height).

4. Final Answer:

   centers = (boxes[:, :2] + boxes[:, 2:]) / 2 sizes = boxes[:, 2:] - boxes[:, :2] converted = torch.cat([centers, sizes], dim=1) -> Option B

5. Quick Check:

   Center = avg edges, size = difference edges [OK]

Hint: Center = average edges; size = difference edges [OK]

Common Mistakes:

• Swapping center and size calculations
• Using addition instead of subtraction for sizes
• Concatenating in wrong order