ReduceLROnPlateau in PyTorch - ML Experiment: Train & Evaluate

Experiment - ReduceLROnPlateau

Problem:You have trained a neural network on a classification task. The training loss decreases steadily, but the validation loss stops improving after some epochs, causing the validation accuracy to plateau around 75%.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%, Validation loss: 0.65

Issue:The learning rate is fixed and too high, causing the model to stop improving on validation data and get stuck at a plateau.

Your Task

Use ReduceLROnPlateau to reduce the learning rate when validation loss stops improving, aiming to increase validation accuracy above 80% without losing training accuracy.

Keep the model architecture unchanged.

Only modify the training loop to include ReduceLROnPlateau scheduler.

Do not change the optimizer type.

Hint 1

Hint 2

Hint 3

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Simple model
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(20, 50)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(50, 2)
    def forward(self, x):
        x = self.relu(self.fc1(x))
        return self.fc2(x)

# Generate dummy data
X_train = torch.randn(500, 20)
y_train = torch.randint(0, 2, (500,))
X_val = torch.randn(100, 20)
y_val = torch.randint(0, 2, (100,))

train_ds = TensorDataset(X_train, y_train)
val_ds = TensorDataset(X_val, y_val)
train_dl = DataLoader(train_ds, batch_size=32, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=32)

model = SimpleNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Add ReduceLROnPlateau scheduler
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3, verbose=True)

epochs = 20
for epoch in range(1, epochs+1):
    model.train()
    for xb, yb in train_dl:
        optimizer.zero_grad()
        preds = model(xb)
        loss = criterion(preds, yb)
        loss.backward()
        optimizer.step()

    model.eval()
    val_loss = 0
    correct = 0
    total = 0
    with torch.no_grad():
        for xb, yb in val_dl:
            preds = model(xb)
            loss_val = criterion(preds, yb)
            val_loss += loss_val.item() * xb.size(0)
            predicted = preds.argmax(dim=1)
            correct += (predicted == yb).sum().item()
            total += yb.size(0)
    val_loss /= total
    val_acc = correct / total * 100

    # Step scheduler with validation loss
    scheduler.step(val_loss)

    print(f"Epoch {epoch}: Val Loss={val_loss:.4f}, Val Acc={val_acc:.2f}%, LR={optimizer.param_groups[0]['lr']:.5f}")

Added torch.optim.lr_scheduler.ReduceLROnPlateau scheduler to reduce learning rate when validation loss plateaus.

Called scheduler.step(val_loss) after each validation phase to monitor validation loss.

Set patience=3 and factor=0.5 to reduce learning rate by half after 3 epochs without improvement.

Results Interpretation

Before: Training accuracy: 92%, Validation accuracy: 75%, Validation loss: 0.65

After: Training accuracy: 90%, Validation accuracy: 82%, Validation loss: 0.48

Using ReduceLROnPlateau helps the model escape plateaus by lowering the learning rate when validation loss stops improving, which improves validation accuracy and reduces overfitting.

Bonus Experiment

Try using a different scheduler like CosineAnnealingLR and compare validation accuracy and loss.

💡 Hint

CosineAnnealingLR changes learning rate smoothly over epochs; adjust its parameters to fit your training length.

Practice

1. What is the main purpose of ReduceLROnPlateau in PyTorch training?

easy

A. To shuffle the training data before each epoch

B. To increase the batch size automatically during training

C. To stop training early when accuracy reaches a threshold

D. To reduce the learning rate when a monitored metric stops improving

Solution

1. Step 1: Understand the role of learning rate schedulers

   Learning rate schedulers adjust the learning rate during training to improve convergence.

2. Step 2: Identify what ReduceLROnPlateau does

   This scheduler reduces the learning rate when a monitored metric (like validation loss) stops improving.

3. Final Answer:

   To reduce the learning rate when a monitored metric stops improving -> Option D

4. Quick Check:

   ReduceLROnPlateau lowers LR on no improvement [OK]

Hint: Remember: it lowers LR when progress stalls [OK]

Common Mistakes:

• Confusing it with early stopping
• Thinking it changes batch size
• Assuming it shuffles data

2. Which of the following is the correct way to create a ReduceLROnPlateau scheduler in PyTorch?

easy

A. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min')

B. scheduler = torch.optim.ReduceLROnPlateau(optimizer, mode='max')

C. scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10)

D. scheduler = torch.optim.ReduceLROnPlateau(optimizer, patience=5)

Solution

1. Step 1: Check the correct module and class name

   The correct class is ReduceLROnPlateau inside torch.optim.lr_scheduler.

2. Step 2: Verify the constructor parameters

   It requires the optimizer and optional parameters like mode. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min') uses correct syntax and parameters.

3. Final Answer:

   scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min') -> Option A

4. Quick Check:

   Correct class and module usage [OK]

Hint: Use torch.optim.lr_scheduler.ReduceLROnPlateau with optimizer [OK]

Common Mistakes:

• Using wrong module path
• Confusing with StepLR scheduler
• Missing required optimizer argument

3. Given the code below, what will be the learning rate after the third call to scheduler.step(val_loss) if val_loss values are [0.5, 0.4, 0.4, 0.4] and patience=2?

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', patience=2, factor=0.1)

val_losses = [0.5, 0.4, 0.4, 0.4]
for loss in val_losses:
    scheduler.step(loss)
    print(f"LR: {optimizer.param_groups[0]['lr']}")

medium

A. 0.1

B. 0.01

C. 0.001

D. 0.4

Solution

1. Step 1: Understand patience and when LR reduces

   Patience=2 means LR reduces after 2 epochs with no improvement in monitored metric.

2. Step 2: Analyze val_loss sequence and scheduler calls

   val_loss improves from 0.5 to 0.4 at second call, then stays same (no improvement) for next two calls. LR reduces only after 2 consecutive no improvements, so after the fourth call, not the third.

3. Final Answer:

   0.1 -> Option A

4. Quick Check:

   LR reduces after patience epochs, not before [OK]

Hint: LR changes after patience epochs without improvement [OK]

Common Mistakes:

• Reducing LR immediately on no improvement
• Confusing patience count
• Using val_loss value as LR

4. Identify the error in the following code snippet using ReduceLROnPlateau:

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)

for epoch in range(5):
    train()
    val_loss = validate()
    scheduler.step()

medium

A. Learning rate must be set to 0.1 initially

B. Optimizer should be SGD, not Adam

C. Missing metric argument in scheduler.step() call

D. scheduler.step() should be called before training

Solution

1. Step 1: Check how ReduceLROnPlateau.step() is called

   This scheduler requires the monitored metric (e.g., val_loss) as an argument in step().

2. Step 2: Identify missing argument in code

   The code calls scheduler.step() without passing val_loss, causing an error.

3. Final Answer:

   Missing metric argument in scheduler.step() call -> Option C

4. Quick Check:

   Pass metric to step() for ReduceLROnPlateau [OK]

Hint: Always pass metric to scheduler.step() for ReduceLROnPlateau [OK]

Common Mistakes:

• Calling step() without metric
• Confusing optimizer type
• Wrong order of scheduler call

5. You want to train a model and reduce the learning rate by half if the validation accuracy does not improve for 3 epochs. Which of the following is the correct way to set up ReduceLROnPlateau for this task?

hard

A. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=2.0, patience=3)

B. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=3)

C. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3)

D. scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=2.0, patience=3)

Solution

1. Step 1: Determine the mode based on metric type

   Validation accuracy should be maximized, so mode='max' is correct.

2. Step 2: Set factor and patience correctly

   Factor=0.5 halves the learning rate, patience=3 waits 3 epochs before reducing.

3. Final Answer:

   scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=3) -> Option B

4. Quick Check:

   Maximize accuracy, reduce LR by half after 3 no improvements [OK]

Hint: Use mode='max' for accuracy, factor <1 to reduce LR [OK]

Common Mistakes:

• Using mode='min' for accuracy
• Setting factor > 1 (increases LR)
• Confusing patience value