Learning rate selection in Computer Vision

Introduction

Learning rate controls how fast a model learns. Choosing the right learning rate helps the model learn well without getting stuck or jumping around.

When training a new image classifier and you want it to learn efficiently.

When your model's training loss is not improving or is unstable.

When you want to speed up training without losing accuracy.

When fine-tuning a pre-trained vision model on a new dataset.

When experimenting with different training setups to find the best results.

Syntax

Computer Vision

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

The learning rate is set as lr in the optimizer.

Common optimizers include SGD, Adam, and RMSprop.

Examples

Set learning rate to 0.01 using SGD optimizer.

Computer Vision

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Set learning rate to 0.001 using Adam optimizer.

Computer Vision

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Reduce learning rate by 10 times every 10 epochs.

Computer Vision

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

Sample Model

This code trains a simple CNN on MNIST for one batch using a learning rate of 0.01. It prints the loss to show training progress.

Computer Vision

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

# Simple CNN model for MNIST
class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(10 * 12 * 12, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = self.pool(torch.relu(self.conv1(x)))
        x = x.view(-1, 10 * 12 * 12)
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Load MNIST data
transform = transforms.Compose([transforms.ToTensor()])
train_dataset = datasets.MNIST('.', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)

# Initialize model, loss, optimizer with learning rate 0.01
model = SimpleCNN()
criterion = nn.CrossEntropyLoss()
learning_rate = 0.01
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

# Train for 1 batch
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    if batch_idx == 0:
        print(f'Batch {batch_idx} Loss: {loss.item():.4f}')
        break

Important Notes

Too high learning rate can make training unstable.

Too low learning rate makes training slow.

Try learning rate schedules to improve training.

Summary

Learning rate controls how fast the model updates.

Choose learning rate carefully for good training.

Use optimizers and schedulers to manage learning rate.