TensorFlow Object Detection API - Model Metrics & Evaluation

Metrics & Evaluation - TensorFlow Object Detection API

Which metric matters for TensorFlow Object Detection API and WHY

In object detection, we want to find and correctly label objects in images. The key metric is Mean Average Precision (mAP). It tells us how well the model finds objects and labels them right. mAP combines how many objects the model found (recall) and how many it labeled correctly (precision). This balance is important because we want to find most objects without many mistakes.

Confusion matrix for object detection

Object detection uses a special confusion matrix based on True Positives (TP), False Positives (FP), and False Negatives (FN) for each object class.

Class: "Cat"
+----------------+----------------+----------------+
|                | Predicted Cat  | Predicted Not  |
|                |                | Cat            |
+----------------+----------------+----------------+
| Actual Cat     | TP = 80        | FN = 20        |
+----------------+----------------+----------------+
| Actual Not Cat | FP = 10        | TN = (ignored) |
+----------------+----------------+----------------+

Note: TN (True Negative) is less used in object detection because many background areas exist.
    

Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89

Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80

Precision vs Recall tradeoff with examples

In object detection:

• High Precision: Few false alarms. Good when mistakes are costly, like detecting weapons in security.
• High Recall: Find most objects. Important when missing an object is bad, like spotting cancer cells.

Example: A wildlife camera detecting rare animals should have high recall to not miss any animals, even if some false alarms happen.

What "good" vs "bad" metric values look like

For TensorFlow Object Detection API models:

• Good mAP: Above 0.7 means the model finds and labels objects well.
• Bad mAP: Below 0.4 means the model misses many objects or labels them wrongly.
• Precision and recall should both be balanced. High precision but very low recall means many objects are missed.

Common pitfalls in metrics

• Accuracy paradox: High accuracy can be misleading if most images have no objects. The model may predict "no object" always and get high accuracy but fail detection.
• Data leakage: If test images are too similar to training, metrics look better but model won't work well in real life.
• Overfitting: Very high training mAP but low test mAP means the model memorizes training images and does not generalize.

Self-check question

Your object detection model has 98% accuracy but only 12% recall on rare objects. Is it good for production?

Answer: No. The high accuracy is misleading because the model misses most rare objects (low recall). For rare object detection, recall is critical to find as many objects as possible. This model needs improvement.

Key Result

Mean Average Precision (mAP) is the key metric balancing precision and recall to evaluate object detection quality.

Practice

1. What is the main purpose of the TensorFlow Object Detection API?

easy

A. To translate languages

B. To generate text from images

C. To find and locate objects in images or videos

D. To create 3D models from photos

Solution

1. Step 1: Understand the API's function

   The TensorFlow Object Detection API is designed to detect objects by finding their locations in images or videos.

2. Step 2: Compare options with the API's purpose

   Only To find and locate objects in images or videos matches this purpose, while others describe unrelated tasks.

3. Final Answer:

   To find and locate objects in images or videos -> Option C

4. Quick Check:

   Object detection = find objects [OK]

Hint: Remember: Object Detection means locating objects [OK]

Common Mistakes:

• Confusing object detection with image translation
• Thinking it generates text from images
• Mixing it up with 3D modeling

2. Which of the following is the correct way to load a pre-trained object detection model in TensorFlow Object Detection API?

easy

A. model = tf.saved_model.load('path_to_model')

B. model = tf.keras.models.load_model('path_to_model')

C. model = tf.load_model('path_to_model')

D. model = tf.object_detection.load('path_to_model')

Solution

1. Step 1: Identify the correct TensorFlow function for loading saved models

   TensorFlow uses tf.saved_model.load() to load saved models, including object detection models.

2. Step 2: Check other options for correctness

   tf.keras.models.load_model() loads Keras models, not specifically object detection saved models; tf.load_model() and tf.object_detection.load() do not exist.

3. Final Answer:

   model = tf.saved_model.load('path_to_model') -> Option A

4. Quick Check:

   Use tf.saved_model.load() for saved models [OK]

Hint: Use tf.saved_model.load() to load detection models [OK]

Common Mistakes:

• Using tf.keras.models.load_model() for detection models
• Trying non-existent functions like tf.load_model()
• Confusing model loading with training functions

3. Given the following code snippet, what will be the output type of detection_boxes after running detection on an image?

import tensorflow as tf
model = tf.saved_model.load('ssd_mobilenet_v2')
image = tf.constant(image_np)
input_tensor = tf.expand_dims(image, 0)
output_dict = model(input_tensor)
detection_boxes = output_dict['detection_boxes']

medium

A. A tensor of shape [1, num_detections, 4] with box coordinates

B. A list of bounding box coordinates as tuples

C. A numpy array of class labels

D. A scalar float value representing confidence

Solution

1. Step 1: Understand the output of TensorFlow Object Detection model

   The model returns a dictionary with keys like 'detection_boxes', which is a tensor containing bounding box coordinates for detected objects.

2. Step 2: Check the shape and type of detection_boxes

   It is a tensor with shape [1, num_detections, 4], where 4 represents box coordinates (ymin, xmin, ymax, xmax).

3. Final Answer:

   A tensor of shape [1, num_detections, 4] with box coordinates -> Option A

4. Quick Check:

   detection_boxes = tensor with box coords [OK]

Hint: Detection boxes are tensors with shape [1, N, 4] [OK]

Common Mistakes:

• Thinking detection_boxes is a list or numpy array
• Confusing boxes with class labels or scores
• Expecting a scalar instead of a tensor

4. You run the following code but get a KeyError: 'detection_scores'. What is the most likely cause?

output_dict = model(input_tensor)
scores = output_dict['detection_scores']

medium

A. The image is not preprocessed

B. The input_tensor shape is incorrect

C. The model was not loaded properly

D. The model output dictionary keys are bytes, not strings

Solution

1. Step 1: Understand the KeyError cause

   A KeyError means the key 'detection_scores' is not found in output_dict.

2. Step 2: Check common output format issue

   TensorFlow Object Detection API sometimes returns keys as bytes (b'detection_scores'), so using string keys causes KeyError.

3. Final Answer:

   The model output dictionary keys are bytes, not strings -> Option D

4. Quick Check:

   Use bytes keys or decode keys to strings [OK]

Hint: Check if output keys are bytes, not strings [OK]

Common Mistakes:

• Assuming input shape error causes KeyError
• Thinking model loading failed without error
• Ignoring key type mismatch in output dictionary

5. You want to improve detection accuracy on a custom dataset using TensorFlow Object Detection API. Which approach is best?

hard

A. Use the model without any training on your data

B. Fine-tune a pre-trained model on your dataset

C. Train a model from scratch with random weights

D. Only increase the number of detection classes without training

Solution

1. Step 1: Understand transfer learning benefits

   Fine-tuning a pre-trained model uses learned features and adapts them to your data, improving accuracy efficiently.

2. Step 2: Evaluate other options

   Training from scratch requires much data and time; using a model without training won't adapt to new data; increasing classes without training won't improve accuracy.

3. Final Answer:

   Fine-tune a pre-trained model on your dataset -> Option B

4. Quick Check:

   Fine-tuning = best for custom data accuracy [OK]

Hint: Fine-tune pre-trained models for better custom results [OK]

Common Mistakes:

• Training from scratch without enough data
• Using pre-trained model without adaptation
• Changing classes without retraining