Mixed precision training (AMP) in PyTorch - Model Pipeline Trace

Model Pipeline - Mixed precision training (AMP)

This pipeline shows how mixed precision training uses both 16-bit and 32-bit numbers to speed up training while keeping accuracy. It helps the model learn faster and use less memory.

Data Flow - 5 Stages

1Data Loading

1000 rows x 28 x 28 pixels→Load images and labels from dataset→1000 rows x 28 x 28 pixels

Image of handwritten digit '5' with label 5

↓

2Preprocessing

1000 rows x 28 x 28 pixels→Normalize pixel values to 0-1 range→1000 rows x 28 x 28 pixels

Pixel value 150 normalized to 0.59

↓

3Feature Engineering

1000 rows x 28 x 28 pixels→Convert images to tensors for model input→1000 rows x 1 x 28 x 28 tensor

Tensor shape for one image: [1, 1, 28, 28]

↓

4Model Training with AMP

Batch of 32 tensors [32, 1, 28, 28]→Train model using mixed precision (float16 and float32) with automatic scaling→Model weights updated, loss scalar

Loss value 0.45 after first batch

↓

5Metrics Calculation

Model predictions and true labels→Calculate accuracy and loss→Accuracy scalar, loss scalar

Accuracy 0.82, Loss 0.45

Training Trace - Epoch by Epoch

Loss
0.7 |*       
0.6 | *      
0.5 |  *     
0.4 |   *    
0.3 |    *   
0.2 |     *  
0.1 |      * 
     --------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.75	Loss starts high, accuracy moderate as model begins learning
2	0.48	0.83	Loss decreases, accuracy improves with mixed precision speeding training
3	0.35	0.89	Model converges faster due to AMP, loss lowers steadily
4	0.28	0.92	Stable training, accuracy nearing high performance
5	0.22	0.94	Final epoch shows good convergence with low loss and high accuracy

Prediction Trace - 6 Layers

Layer 1: Input Layer

Layer 2: Convolutional Layer (float16)

Layer 3: Activation (ReLU)

Layer 4: Fully Connected Layer (float32)

Layer 5: Softmax

Layer 6: Prediction

Model Quiz - 3 Questions

Why does mixed precision training use both float16 and float32 numbers?

A To speed up training and save memory while keeping accuracy

B To make the model smaller but slower

C To increase the model size for better learning

D To avoid using GPUs during training

Key Insight

Mixed precision training balances speed and accuracy by using faster float16 computations where possible and precise float32 where needed. This helps models train faster and use less memory without losing performance.