Warmup strategies in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Warmup strategies

Which metric matters for Warmup strategies and WHY

Warmup strategies help the model start learning smoothly by gradually increasing the learning rate. The key metrics to watch are training loss and validation loss. These show if the model is learning steadily without sudden jumps or getting stuck early. Also, accuracy or other performance metrics on validation data help confirm if warmup improves final results.

Confusion matrix or equivalent visualization

Warmup strategies do not directly affect confusion matrices but influence overall model training stability. A good way to visualize warmup effect is by plotting learning rate over training steps and training/validation loss curves. Smooth loss curves with gradual decrease indicate effective warmup.

Learning Rate Schedule Example:
Step: 0    LR: 0.0001
Step: 100  LR: 0.001
Step: 200  LR: 0.01
Step: 300  LR: 0.1 (max)

Training Loss:
Epoch 1: 0.8
Epoch 2: 0.6
Epoch 3: 0.4

Validation Loss:
Epoch 1: 0.85
Epoch 2: 0.65
Epoch 3: 0.45

Precision vs Recall tradeoff with Warmup strategies

Warmup mainly affects how fast and stable the model learns early on. It does not directly change precision or recall but helps avoid bad early training that can hurt both. For example, without warmup, the model might jump to bad weights causing low recall (missing positives) or low precision (too many false alarms). Warmup helps the model find better balance by starting slow.

Think of warmup like warming up your muscles before exercise. If you start too fast, you might get hurt (bad model). If you warm up well, you perform better overall.

What "good" vs "bad" metric values look like for Warmup strategies

Good warmup: Training and validation loss decrease smoothly from the start. No sudden spikes or jumps. Final accuracy or F1 score is higher compared to no warmup.

Bad warmup or no warmup: Training loss jumps or oscillates early. Validation loss may increase or fluctuate. Final accuracy or F1 score is lower or unstable.

Example:
Good warmup: Training loss steadily drops from 0.8 to 0.3
Bad warmup: Training loss jumps 0.8 -> 1.2 -> 0.9

Metrics pitfalls with Warmup strategies

• Ignoring early loss spikes: Without warmup, early training loss may spike, but ignoring this can hide unstable training.
• Overfitting signs: Warmup helps avoid bad starts, but watch if validation loss rises while training loss falls -- this means overfitting.
• Data leakage: Warmup won't fix data leakage issues that inflate metrics falsely.
• Confusing warmup with learning rate decay: Warmup increases learning rate early, decay reduces it later. Mixing them up can mislead metric interpretation.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, this is not good for fraud detection. The model misses most fraud cases (low recall). Warmup strategies can help training stability but won't fix this imbalance alone. You need to improve recall by adjusting thresholds, using better data, or different loss functions.

Key Result

Warmup strategies improve training stability shown by smooth loss curves and better final accuracy, helping models learn effectively from the start.

Practice

1. What is the main purpose of using a warmup strategy in PyTorch training?

easy

A. To immediately set the learning rate to its maximum value

B. To gradually increase the learning rate at the start of training

C. To decrease the learning rate throughout the entire training

D. To freeze model weights during the first epochs

Solution

1. Step 1: Understand what warmup means

   Warmup means starting with a low learning rate and increasing it slowly.

2. Step 2: Identify the goal of warmup

   This helps the model learn smoothly and avoid sudden big updates that can harm training.

3. Final Answer:

   To gradually increase the learning rate at the start of training -> Option B

4. Quick Check:

   Warmup = gradual learning rate increase [OK]

Hint: Warmup means slowly raising learning rate early [OK]

Common Mistakes:

• Thinking warmup immediately sets max learning rate
• Confusing warmup with learning rate decay
• Assuming warmup freezes model weights

2. Which PyTorch class is commonly used to implement a warmup learning rate schedule with a custom function?

easy

A. torch.optim.lr_scheduler.StepLR

B. torch.optim.lr_scheduler.ReduceLROnPlateau

C. torch.optim.lr_scheduler.LambdaLR

D. torch.optim.lr_scheduler.ExponentialLR

Solution

1. Step 1: Recall PyTorch schedulers for warmup

   LambdaLR allows defining a custom function to adjust learning rate.

2. Step 2: Match scheduler to warmup use

   Warmup needs a custom function to increase learning rate gradually, which LambdaLR supports.

3. Final Answer:

   torch.optim.lr_scheduler.LambdaLR -> Option C

4. Quick Check:

   Custom function scheduler = LambdaLR [OK]

Hint: LambdaLR lets you define custom learning rate changes [OK]

Common Mistakes:

• Choosing StepLR which uses fixed step decay
• Picking ReduceLROnPlateau which reacts to metrics
• Selecting ExponentialLR which decays exponentially

3. Given the following PyTorch code snippet, what will be the learning rate at epoch 3?

import torch
optimizer = torch.optim.SGD([torch.nn.Parameter(torch.randn(2, 2))], lr=0.1)

warmup_epochs = 5
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda epoch: min((epoch + 1) / warmup_epochs, 1))

for epoch in range(5):
    scheduler.step()
    print(f"Epoch {epoch+1} LR: {optimizer.param_groups[0]['lr']}")

medium

A. 0.06

B. 0.03

C. 0.10

D. 0.50

Solution

1. Step 1: Understand the lambda function for LR

   The lambda function returns (epoch+1)/5 until it reaches 1, scaling the base LR 0.1.

2. Step 2: Calculate LR at epoch 3 (0-based index)

   Epoch 3 means epoch=2, so LR factor = (2+1)/5 = 3/5 = 0.6. LR = 0.1 * 0.6 = 0.06.

3. Final Answer:

   0.06 -> Option A

4. Quick Check:

   Epoch 3 LR = 0.1 * 3/5 = 0.06 [OK]

Hint: Multiply base LR by (epoch+1)/warmup_epochs [OK]

Common Mistakes:

• Using epoch number directly without +1
• Confusing epoch index with count
• Assuming LR is constant during warmup

4. Identify the error in this PyTorch warmup scheduler code:

import torch
optimizer = torch.optim.Adam([torch.nn.Parameter(torch.randn(2, 2))], lr=0.01)
warmup_epochs = 3
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda epoch: epoch / warmup_epochs)

for epoch in range(5):
    scheduler.step()
    print(f"Epoch {epoch} LR: {optimizer.param_groups[0]['lr']}")

medium

A. The optimizer should be SGD, not Adam

B. scheduler.step() should be called after optimizer.step()

C. The learning rate is not scaled by base LR

D. The lambda function returns 0 at epoch 0 causing zero LR

Solution

1. Step 1: Analyze lambda function behavior at epoch 0

   At epoch 0, lambda returns 0/3 = 0, so LR is zero, which stops learning initially.

2. Step 2: Understand why zero LR is a problem

   Zero LR means no weight updates, which can slow or stop training progress early.

3. Final Answer:

   The lambda function returns 0 at epoch 0 causing zero LR -> Option D

4. Quick Check:

   Epoch 0 LR = 0 causes no learning [OK]

Hint: Check if lambda returns zero at first epoch [OK]

Common Mistakes:

• Ignoring zero LR at start
• Thinking optimizer type causes error
• Confusing scheduler step order

5. You want to implement a warmup strategy that linearly increases the learning rate from 0 to 0.1 over 4 epochs, then keeps it constant. Which lr_lambda function correctly achieves this in PyTorch's LambdaLR?

hard

A. lambda epoch: min((epoch + 1) / 4, 1)

B. lambda epoch: epoch / 4

C. lambda epoch: 1 if epoch >= 4 else 0.1 * epoch

D. lambda epoch: (epoch + 1) * 0.1

Solution

1. Step 1: Understand the warmup goal

   Learning rate should increase linearly from 0 to 1 (scale factor) over 4 epochs, then stay at 1.

2. Step 2: Check each lambda function

   lambda epoch: min((epoch + 1) / 4, 1) uses min((epoch+1)/4, 1), which linearly increases from 0.25 to 1 by epoch 4, then stays at 1.

3. Final Answer:

   lambda epoch: min((epoch + 1) / 4, 1) -> Option A

4. Quick Check:

   Linear increase capped at 1 = lambda epoch: min((epoch + 1) / 4, 1) [OK]

Hint: Use min((epoch+1)/warmup_epochs, 1) for linear warmup [OK]

Common Mistakes:

• Not adding +1 to epoch causing zero start
• Multiplying by 0.1 inside lambda instead of base LR
• Using step function instead of linear increase