Action recognition basics in Computer Vision - Model Pipeline Trace

Model Pipeline - Action recognition basics

This pipeline learns to recognize human actions from video clips. It processes video frames, extracts important features, trains a model to understand actions, and then predicts the action in new videos.

Data Flow - 5 Stages

1Input Video Frames

100 videos x 30 frames x 64 height x 64 width x 3 channels→Collect raw video frames representing actions→100 videos x 30 frames x 64 height x 64 width x 3 channels

A video showing a person waving, represented as 30 color frames of size 64x64

↓

2Preprocessing

100 videos x 30 frames x 64 height x 64 width x 3 channels→Normalize pixel values to range 0-1 and resize frames if needed→100 videos x 30 frames x 64 height x 64 width x 3 channels

Pixel values changed from 0-255 to 0.0-1.0 for better model training

↓

3Feature Extraction

100 videos x 30 frames x 64 height x 64 width x 3 channels→Use 3D convolution layers to extract motion and appearance features→100 videos x 15 frames x 16 height x 16 width x 64 feature maps

Features capturing movement patterns like waving or jumping

↓

4Temporal Pooling

100 videos x 15 frames x 16 height x 16 width x 64 feature maps→Aggregate features over time to summarize action→100 videos x 64 feature vectors

A single vector representing the whole action in each video

↓

5Classification Model Training

100 videos x 64 feature vectors→Train a neural network classifier to label actions→100 videos x 5 action classes

Labels like 'waving', 'jumping', 'running', 'clapping', 'walking'

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.9 | *      
0.7 |  *     
0.5 |   *    
0.4 |    *   
    +---------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.40	Model starts learning, accuracy is low
2	0.9	0.55	Loss decreases, accuracy improves
3	0.7	0.68	Model captures action features better
4	0.5	0.78	Good improvement, model generalizes well
5	0.4	0.83	Training converges with high accuracy

Prediction Trace - 5 Layers

Layer 1: Input Video Frames

Layer 2: Preprocessing

Layer 3: 3D Convolutional Feature Extraction

Layer 4: Temporal Pooling

Layer 5: Classification Layer

Model Quiz - 3 Questions

What is the main purpose of the 3D convolutional layer in this pipeline?

A To convert video frames into grayscale images

B To capture motion and appearance features across video frames

C To normalize pixel values between 0 and 1

D To split videos into training and testing sets

Key Insight

Action recognition models learn by extracting motion and appearance features from video frames over time. Normalizing inputs and using 3D convolutions help the model understand actions better. Training shows steady improvement in accuracy as the model learns to distinguish different actions.