Video understanding basics in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Video understanding basics

Which metric matters for Video Understanding and WHY

Video understanding means teaching a computer to watch videos and know what is happening. The main goal is to recognize actions, objects, or events in the video.

The key metrics are Accuracy, Precision, Recall, and F1-score. These tell us how well the model identifies the right actions or objects without mistakes.

For example, if the model says "someone is running" in a video, precision tells us how often it was right when it said that. Recall tells us how many times it found all the running moments in the video. F1-score balances both.

Sometimes, we also use Mean Average Precision (mAP) for detecting multiple objects or actions in videos. It measures how well the model finds all correct items and avoids wrong ones.

Confusion Matrix for Video Action Recognition

        Predicted
        Run   Walk   Jump
    Run   50     5      2
    Walk  3      45     4
    Jump  1      2      40

    True labels total = 50+5+2+3+45+4+1+2+40 = 152
    

From this matrix:

• True Positives (TP) for "Run" = 50
• False Positives (FP) for "Run" = 3 + 1 = 4 (times model said "Run" but it was "Walk" or "Jump")
• False Negatives (FN) for "Run" = 5 + 2 = 7 (times model missed "Run")
• True Negatives (TN) = total - TP - FP - FN = 152 - 50 - 4 - 7 = 91

We use these to calculate precision, recall, and F1 for each action.

Precision vs Recall Tradeoff in Video Understanding

Imagine a security camera that detects suspicious actions like "falling".

• High Precision: When the model says "falling", it is almost always correct. This avoids false alarms.
• High Recall: The model finds almost every actual fall, even if it sometimes makes mistakes.

If we want to avoid missing any falls (important for safety), we focus on high recall.

If we want to avoid bothering people with false alarms, we focus on high precision.

Balancing both with F1-score helps find a good middle ground.

What Good vs Bad Metrics Look Like for Video Understanding

Good metrics:

• Accuracy above 85% on recognizing actions
• Precision and recall both above 80%
• F1-score close to precision and recall, showing balance
• Mean Average Precision (mAP) above 75% for object detection in videos

Bad metrics:

• High accuracy but very low recall (model misses many actions)
• High recall but very low precision (model makes many false detections)
• F1-score much lower than precision or recall, showing imbalance
• mAP below 50%, meaning poor detection quality

Common Pitfalls in Video Understanding Metrics

• Accuracy Paradox: Videos often have many frames with no action. A model that always says "no action" can have high accuracy but is useless.
• Data Leakage: Using parts of the same video in training and testing can make metrics look better than reality.
• Overfitting: Model performs well on training videos but poorly on new videos, showing metrics that don't generalize.
• Ignoring Class Imbalance: Some actions happen rarely. Metrics must consider this to avoid misleading results.

Self-Check Question

Your video action recognition model has 98% accuracy but only 12% recall on "falling" events. Is it good for safety monitoring? Why or why not?

Answer: No, it is not good. The model misses 88% of actual falls (low recall), which is dangerous for safety. High accuracy is misleading because most frames have no falls. Improving recall is critical here.

Key Result

In video understanding, balanced precision and recall (measured by F1-score) are key to reliable action detection, not just accuracy.

Practice

1. What is the main goal of video understanding in AI?

easy

A. Teaching computers to watch and learn from videos

B. Making videos play faster on devices

C. Compressing videos to save space

D. Editing videos automatically

Solution

1. Step 1: Understand the purpose of video understanding

   Video understanding means enabling computers to analyze and learn from video content.

2. Step 2: Compare options to the definition

   Only Teaching computers to watch and learn from videos matches this goal; others relate to video playback, compression, or editing.

3. Final Answer:

   Teaching computers to watch and learn from videos -> Option A

4. Quick Check:

   Video understanding = Teaching computers to learn from videos [OK]

Hint: Focus on learning, not playback or editing [OK]

Common Mistakes:

• Confusing video understanding with video editing
• Thinking it's about video compression
• Assuming it's about video playback speed

2. Which neural network type is commonly used for video understanding?

easy

A. Fully connected networks without convolution

B. 2D convolutional neural networks

C. Recurrent neural networks only

D. 3D convolutional neural networks

Solution

1. Step 1: Identify network types used for video data

   Videos have spatial and temporal dimensions; 3D CNNs capture both.

2. Step 2: Match network type to video understanding

   3D CNNs process frames over time, unlike 2D CNNs or fully connected nets.

3. Final Answer:

   3D convolutional neural networks -> Option D

4. Quick Check:

   3D CNNs capture space and time in videos [OK]

Hint: Remember 3D CNNs handle time and space in videos [OK]

Common Mistakes:

• Choosing 2D CNNs which only see single frames
• Ignoring temporal info by picking fully connected nets
• Assuming RNNs alone are best for video frames

3. Given this Python snippet for video data preprocessing, what is the shape of the output tensor?

import numpy as np
video = np.random.rand(16, 64, 64, 3)  # 16 frames, 64x64 size, 3 color channels
output = video.reshape(1, 16, 64, 64, 3)

medium

A. (16, 64, 64, 3)

B. (64, 64, 3, 16)

C. (1, 16, 64, 64, 3)

D. (16, 1, 64, 64, 3)

Solution

1. Step 1: Understand the original video shape

   The video has shape (16, 64, 64, 3): 16 frames, each 64x64 pixels with 3 color channels.

2. Step 2: Analyze the reshape operation

   Reshape adds a new dimension at the front, making shape (1, 16, 64, 64, 3).

3. Final Answer:

   (1, 16, 64, 64, 3) -> Option C

4. Quick Check:

   Reshape adds batch dimension = (1, 16, 64, 64, 3) [OK]

Hint: Look for added batch dimension in reshape [OK]

Common Mistakes:

• Ignoring the added batch dimension
• Mixing up order of dimensions
• Assuming reshape changes total elements

4. This code snippet tries to create a 3D CNN layer but has an error. What is the mistake?

from tensorflow.keras.layers import Conv3D
layer = Conv3D(filters=32, kernel_size=(3,3), activation='relu')

medium

A. kernel_size should have three dimensions, e.g., (3,3,3)

B. Missing input shape argument

C. filters must be a list, not an integer

D. activation='relu' is not allowed in Conv3D

Solution

1. Step 1: Check Conv3D kernel_size parameter

   Conv3D expects a 3D kernel size tuple for depth, height, width.

2. Step 2: Identify the error in kernel_size

   The code uses (3,3), missing the third dimension, causing an error.

3. Final Answer:

   kernel_size should have three dimensions, e.g., (3,3,3) -> Option A

4. Quick Check:

   3D CNN kernel_size needs 3 values [OK]

Hint: 3D kernels need three numbers, not two [OK]

Common Mistakes:

• Using 2D kernel size in 3D CNN
• Thinking filters must be a list
• Believing activation can't be relu

5. You want to train a video understanding model to recognize actions. Which data setup is best?

hard

A. Single images with labels, no temporal info

B. Video clips with labels and enough frames to see actions

C. Random frames from different videos without labels

D. Audio clips extracted from videos

Solution

1. Step 1: Understand training data needs for action recognition

   Actions happen over time, so clips with multiple frames are needed.

2. Step 2: Evaluate options for temporal and label info

   Only Video clips with labels and enough frames to see actions provides labeled video clips with enough frames to capture actions.

3. Final Answer:

   Video clips with labels and enough frames to see actions -> Option B

4. Quick Check:

   Training needs labeled clips with temporal info [OK]

Hint: Actions need multiple frames with labels [OK]

Common Mistakes:

• Using single images without time info
• Ignoring labels in training data
• Using unrelated audio clips