StepLR and MultiStepLR in PyTorch - Model Pipeline Trace

Model Pipeline - StepLR and MultiStepLR

This pipeline shows how learning rate schedulers StepLR and MultiStepLR adjust the learning rate during training to help the model learn better and faster.

Data Flow - 5 Stages

1Data Loading

1000 rows x 10 features→Load dataset with 10 features per sample→1000 rows x 10 features

[[0.5, 1.2, ..., 0.3], [0.1, 0.4, ..., 0.7], ...]

↓

2Model Initialization

1000 rows x 10 features→Initialize a simple neural network with 10 input nodes and 2 output nodes→Model ready for training

Neural network with layers: Input(10) -> Hidden(5) -> Output(2)

↓

3Optimizer Setup

Model parameters→Set optimizer with initial learning rate 0.1→Optimizer ready

SGD optimizer with lr=0.1

↓

4Learning Rate Scheduler Setup

Optimizer with lr=0.1→Apply StepLR or MultiStepLR scheduler to adjust learning rate during training→Scheduler ready to update learning rate

StepLR with step_size=5, gamma=0.5 or MultiStepLR with milestones=[3,7], gamma=0.1

↓

5Training Loop

Training data and model→Train model for 10 epochs, update learning rate each epoch using scheduler→Trained model with updated learning rates

Epoch 1 lr=0.1, Epoch 5 lr=0.05 (StepLR), Epoch 3 lr=0.01 (MultiStepLR)

Training Trace - Epoch by Epoch

Loss
1.0 |*
0.9  | *
0.8  |  *
0.7  |   *
0.6  |    *
0.5  |     *
0.4  |      *
0.3  |       *
     +----------------
      1 2 3 4 5 6 7 8 9 10 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.60	Initial training with learning rate 0.1
2	0.70	0.68	Loss decreased, accuracy improved
3	0.60	0.72	Learning rate unchanged for StepLR, decreased for MultiStepLR
4	0.55	0.75	Model continues to improve
5	0.50	0.78	StepLR reduces learning rate by gamma=0.5 here
6	0.45	0.80	Lower learning rate helps fine-tune weights
7	0.42	0.82	MultiStepLR reduces learning rate at this milestone
8	0.40	0.83	Training stabilizes with smaller learning rate
9	0.38	0.84	Model converges further
10	0.36	0.85	Final epoch with lowest learning rate

Prediction Trace - 3 Layers

Layer 1: Input Layer

Layer 2: Hidden Layer (ReLU)

Layer 3: Output Layer (Softmax)

Model Quiz - 3 Questions

What does the StepLR scheduler do during training?

A Reduces learning rate by a factor after fixed number of epochs

B Increases learning rate gradually every epoch

C Keeps learning rate constant throughout training

D Randomly changes learning rate each epoch

Key Insight

Learning rate schedulers like StepLR and MultiStepLR help the model train better by reducing the learning rate at strategic points, allowing faster learning early on and finer adjustments later to improve accuracy and reduce loss.