Video understanding basics in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Video understanding basics

Problem:We want to teach a computer to understand simple actions in short videos, like recognizing if someone is walking or running.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting: it performs very well on training data but poorly on new, unseen videos.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change the model architecture and training settings.

Do not change the dataset or add more data.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.callbacks import EarlyStopping

# Sample video data shape: (num_samples, frames, height, width, channels)
# For simplicity, we simulate data here
import numpy as np

num_samples = 1000
frames = 10
height = 64
width = 64
channels = 3
num_classes = 2

X_train = np.random.rand(num_samples, frames, height, width, channels).astype('float32')
y_train = np.random.randint(0, num_classes, size=(num_samples,))

X_val = np.random.rand(200, frames, height, width, channels).astype('float32')
y_val = np.random.randint(0, num_classes, size=(200,))

# Build a simple 3D CNN model with dropout
model = models.Sequential([
    layers.Conv3D(32, kernel_size=(3,3,3), activation='relu', input_shape=(frames, height, width, channels)),
    layers.MaxPooling3D(pool_size=(1,2,2)),
    layers.Dropout(0.3),
    layers.Conv3D(64, kernel_size=(3,3,3), activation='relu'),
    layers.MaxPooling3D(pool_size=(2,2,2)),
    layers.Dropout(0.3),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.4),
    layers.Dense(num_classes, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=50, batch_size=32,
                    validation_data=(X_val, y_val), callbacks=[early_stop])

Added dropout layers after convolution and dense layers to reduce overfitting.

Lowered learning rate from default to 0.0005 for smoother training.

Added early stopping to halt training when validation loss stops improving.

Results Interpretation

Before: Training accuracy was 95%, validation accuracy was 70%, showing overfitting.

After: Training accuracy reduced to 90%, validation accuracy improved to 87%, indicating better generalization.

Adding dropout and early stopping helps the model avoid memorizing training data and perform better on new videos. Lower learning rate helps the model learn more carefully.

Bonus Experiment

Try using a pretrained video model like MobileNet3D or I3D and fine-tune it on this dataset to improve accuracy further.

💡 Hint

Pretrained models have learned useful features from large video datasets and can help your model understand videos better with less training.

Practice

1. What is the main goal of video understanding in AI?

easy

A. Teaching computers to watch and learn from videos

B. Making videos play faster on devices

C. Compressing videos to save space

D. Editing videos automatically

Solution

1. Step 1: Understand the purpose of video understanding

   Video understanding means enabling computers to analyze and learn from video content.

2. Step 2: Compare options to the definition

   Only Teaching computers to watch and learn from videos matches this goal; others relate to video playback, compression, or editing.

3. Final Answer:

   Teaching computers to watch and learn from videos -> Option A

4. Quick Check:

   Video understanding = Teaching computers to learn from videos [OK]

Hint: Focus on learning, not playback or editing [OK]

Common Mistakes:

• Confusing video understanding with video editing
• Thinking it's about video compression
• Assuming it's about video playback speed

2. Which neural network type is commonly used for video understanding?

easy

A. Fully connected networks without convolution

B. 2D convolutional neural networks

C. Recurrent neural networks only

D. 3D convolutional neural networks

Solution

1. Step 1: Identify network types used for video data

   Videos have spatial and temporal dimensions; 3D CNNs capture both.

2. Step 2: Match network type to video understanding

   3D CNNs process frames over time, unlike 2D CNNs or fully connected nets.

3. Final Answer:

   3D convolutional neural networks -> Option D

4. Quick Check:

   3D CNNs capture space and time in videos [OK]

Hint: Remember 3D CNNs handle time and space in videos [OK]

Common Mistakes:

• Choosing 2D CNNs which only see single frames
• Ignoring temporal info by picking fully connected nets
• Assuming RNNs alone are best for video frames

3. Given this Python snippet for video data preprocessing, what is the shape of the output tensor?

import numpy as np
video = np.random.rand(16, 64, 64, 3)  # 16 frames, 64x64 size, 3 color channels
output = video.reshape(1, 16, 64, 64, 3)

medium

A. (16, 64, 64, 3)

B. (64, 64, 3, 16)

C. (1, 16, 64, 64, 3)

D. (16, 1, 64, 64, 3)

Solution

1. Step 1: Understand the original video shape

   The video has shape (16, 64, 64, 3): 16 frames, each 64x64 pixels with 3 color channels.

2. Step 2: Analyze the reshape operation

   Reshape adds a new dimension at the front, making shape (1, 16, 64, 64, 3).

3. Final Answer:

   (1, 16, 64, 64, 3) -> Option C

4. Quick Check:

   Reshape adds batch dimension = (1, 16, 64, 64, 3) [OK]

Hint: Look for added batch dimension in reshape [OK]

Common Mistakes:

• Ignoring the added batch dimension
• Mixing up order of dimensions
• Assuming reshape changes total elements

4. This code snippet tries to create a 3D CNN layer but has an error. What is the mistake?

from tensorflow.keras.layers import Conv3D
layer = Conv3D(filters=32, kernel_size=(3,3), activation='relu')

medium

A. kernel_size should have three dimensions, e.g., (3,3,3)

B. Missing input shape argument

C. filters must be a list, not an integer

D. activation='relu' is not allowed in Conv3D

Solution

1. Step 1: Check Conv3D kernel_size parameter

   Conv3D expects a 3D kernel size tuple for depth, height, width.

2. Step 2: Identify the error in kernel_size

   The code uses (3,3), missing the third dimension, causing an error.

3. Final Answer:

   kernel_size should have three dimensions, e.g., (3,3,3) -> Option A

4. Quick Check:

   3D CNN kernel_size needs 3 values [OK]

Hint: 3D kernels need three numbers, not two [OK]

Common Mistakes:

• Using 2D kernel size in 3D CNN
• Thinking filters must be a list
• Believing activation can't be relu

5. You want to train a video understanding model to recognize actions. Which data setup is best?

hard

A. Single images with labels, no temporal info

B. Video clips with labels and enough frames to see actions

C. Random frames from different videos without labels

D. Audio clips extracted from videos

Solution

1. Step 1: Understand training data needs for action recognition

   Actions happen over time, so clips with multiple frames are needed.

2. Step 2: Evaluate options for temporal and label info

   Only Video clips with labels and enough frames to see actions provides labeled video clips with enough frames to capture actions.

3. Final Answer:

   Video clips with labels and enough frames to see actions -> Option B

4. Quick Check:

   Training needs labeled clips with temporal info [OK]

Hint: Actions need multiple frames with labels [OK]

Common Mistakes:

• Using single images without time info
• Ignoring labels in training data
• Using unrelated audio clips