StepLR and MultiStepLR in PyTorch

Introduction

StepLR and MultiStepLR help adjust the learning speed of a model during training. This makes the model learn better and faster by lowering the learning rate at certain points.

When training a model and you want to reduce the learning rate after some fixed number of steps to improve learning.

When you want to lower the learning rate at specific milestones during training to help the model settle into better solutions.

When your training loss stops improving and you want to slow down learning to fine-tune the model.

When you want to avoid the model jumping around too much by gradually reducing the learning rate.

Syntax

PyTorch

torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)
torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)

StepLR reduces the learning rate every step_size epochs by multiplying it by gamma.

MultiStepLR reduces the learning rate at specific epochs listed in milestones.

Examples

Reduces learning rate by half every 5 epochs.

PyTorch

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)

Reduces learning rate by 10 times at epochs 3, 7, and 10.

PyTorch

scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[3, 7, 10], gamma=0.1)

Sample Model

This code shows how StepLR reduces the learning rate every 3 epochs by half. You see the learning rate and loss printed each epoch.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple model
model = nn.Linear(2, 1)

# Optimizer
optimizer = optim.SGD(model.parameters(), lr=0.1)

# StepLR scheduler: reduce LR by 0.5 every 3 epochs
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.5)

# Training loop simulation for 10 epochs
for epoch in range(1, 11):
    # Dummy training step
    optimizer.zero_grad()
    inputs = torch.tensor([[1.0, 2.0]])
    target = torch.tensor([[1.0]])
    output = model(inputs)
    loss = (output - target).pow(2).mean()
    loss.backward()
    optimizer.step()

    # Step the scheduler
    scheduler.step()

    # Print current learning rate
    current_lr = optimizer.param_groups[0]['lr']
    print(f"Epoch {epoch}: Learning Rate = {current_lr:.5f}, Loss = {loss.item():.5f}")

Important Notes

Always call scheduler.step() after optimizer.step() in each epoch.

Learning rate changes help the model avoid overshooting the best solution.

StepLR is simpler; MultiStepLR gives more control with specific milestones.

Summary

StepLR and MultiStepLR adjust learning rate during training to improve model learning.

StepLR reduces learning rate every fixed number of epochs.

MultiStepLR reduces learning rate at chosen epochs.