Learning rate schedulers in PyTorch - Model Pipeline Trace

Model Pipeline - Learning rate schedulers

This pipeline shows how a learning rate scheduler adjusts the learning rate during training to help the model learn better and faster. It starts with data, trains a model, and changes the learning rate step-by-step to improve accuracy.

Data Flow - 4 Stages

1Data Loading

1000 rows x 10 columns→Load dataset with 10 features per sample→1000 rows x 10 columns

[[0.5, 1.2, ..., 0.3], [0.1, 0.4, ..., 0.9], ...]

↓

2Train/Test Split

1000 rows x 10 columns→Split data into 800 training and 200 testing samples→800 rows x 10 columns (train), 200 rows x 10 columns (test)

Train sample: [0.5, 1.2, ..., 0.3], Test sample: [0.2, 0.7, ..., 0.1]

↓

3Model Initialization

800 rows x 10 columns→Initialize neural network with input size 10 and output size 2→Model with parameters (weights and biases)

Layer1 weights shape: (10, 10), Layer2 weights shape: (10, 2)

↓

4Training with Scheduler

800 rows x 10 columns→Train model with learning rate scheduler adjusting learning rate every 5 epochs→Trained model with updated parameters

Learning rate starts at 0.1, reduces to 0.05 at epoch 5, then 0.025 at epoch 10

Training Trace - Epoch by Epoch

Loss
0.7 |*       
0.6 | **     
0.5 |  ***   
0.4 |    ****
0.3 |      ****
0.2 |        ***
    +---------
     1 2 3 4 5 6 7 8 9 10 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Initial training with learning rate 0.1
2	0.55	0.68	Loss decreased, accuracy improved
3	0.48	0.73	Model learning well with current learning rate
4	0.42	0.77	Steady improvement
5	0.38	0.80	Learning rate reduced to 0.05 by scheduler
6	0.35	0.82	Lower learning rate helps fine-tune weights
7	0.32	0.84	Continued improvement
8	0.30	0.85	Model converging
9	0.28	0.86	Stable training
10	0.26	0.87	Learning rate reduced to 0.025 by scheduler

Prediction Trace - 3 Layers

Layer 1: Input Layer

Layer 2: Hidden Layer with ReLU

Layer 3: Output Layer with Softmax

Model Quiz - 3 Questions

What happens to the learning rate at epoch 5?

A It decreases to half the initial value

B It increases to speed up training

C It stays the same

D It becomes zero

Key Insight

Learning rate schedulers help the model start learning quickly with a higher rate, then slow down to fine-tune weights. This balance improves accuracy and helps the model converge smoothly.