Frame extraction in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Frame extraction

Which metric matters for Frame Extraction and WHY

Frame extraction means picking important images from a video. The main goal is to get frames that best represent the video content without missing key moments or adding too many similar frames.

Metrics to check how well this works include:

• Precision: How many extracted frames are actually important (not duplicates or irrelevant)?
• Recall: How many important frames did we manage to extract out of all important frames?
• F1 Score: A balance between precision and recall to see overall quality.
• Frame Rate Consistency: Ensures frames are extracted evenly or at meaningful changes.

We want high recall to not miss key frames, and high precision to avoid too many useless frames.

Confusion Matrix for Frame Extraction

Imagine we label frames as "Important" or "Not Important". The confusion matrix counts:

      |---------------------------|
      |           | Predicted     |
      | Actual    | Important | Not Important |
      |-----------|-----------|--------------|
      | Important |    TP     |      FN      |
      | Not Imp.  |    FP     |      TN      |
      |---------------------------|
    

Where:

• TP (True Positive): Important frames correctly extracted.
• FP (False Positive): Not important frames wrongly extracted.
• FN (False Negative): Important frames missed.
• TN (True Negative): Not important frames correctly ignored.

Precision vs Recall Tradeoff in Frame Extraction

If you extract too many frames, you get high recall but low precision (many unimportant frames included).

If you extract very few frames, you get high precision but low recall (missing important frames).

Example:

• Extracting every frame: Recall = 100%, Precision = low (lots of duplicates).
• Extracting only very few frames: Precision = high, Recall = low (miss key moments).

Good frame extraction balances both to get meaningful frames without overload.

What Good vs Bad Metrics Look Like for Frame Extraction

• Good: Precision and Recall both above 80%, F1 score high, frames well represent video content.
• Bad: Precision below 50% (too many useless frames), or Recall below 50% (missing key frames).
• Too many extracted frames (high FP) means wasted storage and processing.
• Too few extracted frames (high FN) means loss of important information.

Common Pitfalls in Frame Extraction Metrics

• Accuracy Paradox: If most frames are not important, a naive method that extracts none can have high accuracy but zero recall.
• Data Leakage: Using future frames to decide current frame extraction can give unrealistically good metrics.
• Overfitting: Extracting frames that only fit training videos but fail on new videos.
• Ignoring Temporal Context: Extracting frames without considering video flow can miss important changes.

Self Check: Your model has 98% accuracy but 12% recall on important frames. Is it good?

No, it is not good. The high accuracy likely comes from correctly ignoring many unimportant frames (TN), but the very low recall means it misses most important frames. This defeats the purpose of frame extraction, which is to capture key frames. You should improve recall even if accuracy drops.

Key Result

High recall and precision together ensure frame extraction captures key frames without too many extras.