Learning rate differential in PyTorch - Model Pipeline Trace

Model Pipeline - Learning rate differential

This pipeline shows how using different learning rates for different parts of a neural network helps the model learn better. It trains a simple model with two layers, each having its own learning rate.

Data Flow - 6 Stages

1Data in

1000 rows x 10 columns→Synthetic dataset creation with 10 features and binary labels→1000 rows x 10 columns

[[0.5, -1.2, 0.3, ..., 0.1], label=1]

↓

2Preprocessing

1000 rows x 10 columns→Normalize features to zero mean and unit variance→1000 rows x 10 columns

[[0.0, -0.8, 0.2, ..., 0.05], label=1]

↓

3Feature Engineering

1000 rows x 10 columns→No additional features added→1000 rows x 10 columns

[[0.0, -0.8, 0.2, ..., 0.05], label=1]

↓

4Model Trains

1000 rows x 10 columns→Train 2-layer neural network with differential learning rates: 0.01 for first layer, 0.001 for second layer→Model weights updated

Layer1 weights shape: (5, 10), Layer2 weights shape: (1, 5)

↓

5Metrics Improve

Training epochs→Loss decreases and accuracy increases over 10 epochs→Final loss ~0.25, accuracy ~88%

Epoch 10: loss=0.25, accuracy=0.88

↓

6Prediction

1 row x 10 columns→Model predicts probability for binary class→1 row x 1 column (probability)

[0.85]

Training Trace - Epoch by Epoch

Loss
1.0 | *
0.9 |  *
0.8 |   *
0.7 |    *
0.6 |     *
0.5 |      *
0.4 |       *
0.3 |        *
0.2 |         *
    +----------------
     1 2 3 4 5 6 7 8 9 10 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.55	High loss and low accuracy at start
2	0.65	0.68	Loss decreases, accuracy improves
3	0.50	0.75	Model learns important patterns
4	0.40	0.80	Steady improvement
5	0.35	0.83	Learning rate differential helps stabilize training
6	0.30	0.85	Loss continues to decrease
7	0.28	0.86	Accuracy improves slowly
8	0.27	0.87	Model converging
9	0.26	0.87	Small improvements
10	0.25	0.88	Training stabilizes with good accuracy

Prediction Trace - 5 Layers

Layer 1: Input Layer

Layer 2: First Layer (learning rate 0.01)

Layer 3: Activation (ReLU)

Layer 4: Second Layer (learning rate 0.001)

Layer 5: Sigmoid Activation

Model Quiz - 3 Questions

Why use different learning rates for different layers?

A To make training slower overall

B To allow faster learning in some layers and stable updates in others

C To reduce the number of layers

D To increase the model size

Key Insight

Using different learning rates for different layers helps the model learn faster in some parts while keeping other parts stable. This balance improves training speed and final accuracy.