Gradient accumulation in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Gradient accumulation

Which metric matters for Gradient Accumulation and WHY

Gradient accumulation helps when training with small batches due to memory limits. The key metric to watch is training loss over time. This shows if the model is learning well despite smaller batch updates. Also, validation loss and accuracy help confirm if the model generalizes well. Gradient accumulation aims to mimic a larger batch size by summing gradients over steps before updating weights, so metrics should behave similarly to training with large batches.

Confusion matrix or equivalent visualization

Gradient accumulation itself does not change the confusion matrix directly. However, the confusion matrix after training with gradient accumulation should be similar to training with a large batch size. For example, if a binary classifier has:

      |                    | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|--------------------|
      | Actual Positive     | True Positive (TP)  | False Negative (FN) |
      | Actual Negative     | False Positive (FP) | True Negative (TN)  |
    

After training with gradient accumulation, these counts should be close to those from training with a large batch size, indicating similar model performance.

Precision vs Recall tradeoff with Gradient Accumulation

Gradient accumulation affects how often weights update, which can influence model convergence speed and stability. If accumulation steps are too large, updates become less frequent, possibly slowing learning and affecting precision and recall.

For example, in a spam filter:

• High precision means fewer good emails marked as spam.
• High recall means catching most spam emails.

If gradient accumulation causes unstable training, precision or recall might drop. Finding the right accumulation steps balances memory limits and model quality.

What "good" vs "bad" metric values look like for Gradient Accumulation

Good:

• Training loss steadily decreases over epochs.
• Validation loss decreases or stabilizes without big jumps.
• Accuracy, precision, and recall close to those from training with large batch size.
• Confusion matrix shows balanced TP, TN, FP, FN similar to baseline.

Bad:

• Training loss fluctuates or does not decrease.
• Validation loss increases, indicating overfitting or unstable updates.
• Accuracy, precision, or recall much worse than baseline large batch training.
• Confusion matrix shows many false positives or false negatives.

Common pitfalls with metrics and Gradient Accumulation

• Ignoring effective batch size: Forgetting to multiply accumulation steps by batch size can mislead about training stability.
• Confusing loss scale: Loss values might appear different if not averaged properly over accumulation steps.
• Overfitting signs: If validation loss rises while training loss falls, model might be overfitting despite accumulation.
• Data leakage: Metrics can be falsely high if validation data leaks into training.
• Delayed updates: Large accumulation steps delay weight updates, which can cause slower convergence or unstable metrics.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. The high accuracy likely comes from many normal (non-fraud) cases being correctly classified, but the very low recall means the model misses 88% of fraud cases. For fraud detection, recall is critical because missing fraud is costly. Gradient accumulation should not hide such problems; you must check recall and other metrics, not just accuracy.

Key Result

Gradient accumulation aims to match large batch training metrics like loss and accuracy by summing gradients over steps, but watch training stability and recall especially.

Practice

1. What is the main purpose of gradient accumulation in PyTorch training?

easy

A. To reduce the number of model parameters

B. To increase the learning rate automatically

C. To change the model architecture dynamically

D. To split a large batch into smaller steps to save memory

Solution

1. Step 1: Understand gradient accumulation concept

   Gradient accumulation splits a large batch into smaller mini-batches to avoid memory overflow.

2. Step 2: Identify the main benefit

   This technique helps save GPU memory by processing smaller batches sequentially before updating weights.

3. Final Answer:

   To split a large batch into smaller steps to save memory -> Option D

4. Quick Check:

   Gradient accumulation = batch splitting for memory saving [OK]

Hint: Gradient accumulation saves memory by splitting batches [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with model size reduction
• Assuming it alters model layers

2. Which PyTorch code snippet correctly implements gradient accumulation over 4 steps?

easy

A. for i, data in enumerate(dataloader):
outputs = model(data)
loss = criterion(outputs, labels)
optimizer.step()
loss.backward()
optimizer.zero_grad()

B. for i, data in enumerate(dataloader):
outputs = model(data)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
if (i + 1) % 4 == 0:
optimizer.zero_grad()

C. for i, data in enumerate(dataloader):
outputs = model(data)
loss = criterion(outputs, labels)
loss.backward()
if (i + 1) % 4 == 0:
optimizer.step()
optimizer.zero_grad()

D. for i, data in enumerate(dataloader):
outputs = model(data)
loss = criterion(outputs, labels)
optimizer.zero_grad()
loss.backward()
if (i + 1) % 4 == 0:
optimizer.step()

Solution

1. Step 1: Identify correct order of operations

   Gradient accumulation requires calling loss.backward() each step, then optimizer.step() and zero_grad() after accumulating gradients.

2. Step 2: Check modulo condition for accumulation

   Only after 4 mini-batches (i+1 % 4 == 0) should optimizer.step() and zero_grad() be called to update weights and reset gradients.

3. Final Answer:

   Code that calls loss.backward() every step and optimizer.step() plus zero_grad() every 4 steps -> Option C

4. Quick Check:

   Backward every step, step and zero_grad every 4 steps [OK]

Hint: Call backward every step, step and zero_grad every N steps [OK]

Common Mistakes:

• Calling optimizer.step() before loss.backward()
• Resetting gradients too early
• Not using modulo condition correctly

3. Given this PyTorch code with gradient accumulation steps=2, what is the printed loss after 4 batches?

optimizer.zero_grad()
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels) / 2
    loss.backward()
    if (i + 1) % 2 == 0:
        optimizer.step()
        optimizer.zero_grad()
    print(f"Batch {i+1} loss: {loss.item()}")

medium

A. Loss values cause runtime error due to division

B. Loss values divided by 2 printed for each batch

C. Loss values accumulated and printed only every 2 batches

D. Loss values printed without division

Solution

1. Step 1: Understand loss scaling in code

   Loss is divided by 2 before backward to average gradients over 2 accumulation steps.

2. Step 2: Check print statement inside loop

   Loss.item() prints the scaled loss for each batch, so printed values are divided by 2 every batch.

3. Final Answer:

   Loss values divided by 2 printed for each batch -> Option B

4. Quick Check:

   Loss scaled by 2 and printed each batch [OK]

Hint: Loss is scaled before backward, printed as scaled every batch [OK]

Common Mistakes:

• Thinking loss prints only every 2 batches
• Ignoring loss division by 2
• Assuming runtime error from division

4. Identify the error in this gradient accumulation code snippet:

optimizer.zero_grad()
for i, (inputs, labels) in enumerate(dataloader):
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    if i % 4 == 0:
        optimizer.step()
        optimizer.zero_grad()

medium

A. optimizer.step() and zero_grad() are called too early at i=0

B. loss.backward() is missing gradient scaling

C. optimizer.zero_grad() should be called before loss.backward()

D. The loop should use (i + 1) % 4 == 0 instead of i % 4 == 0

Solution

1. Step 1: Analyze the modulo condition

   Condition i % 4 == 0 triggers optimizer.step() and zero_grad() at i=0, which is before any gradients accumulate.

2. Step 2: Understand correct timing for optimizer step

   Optimizer step should happen after 4 batches, so condition should be (i + 1) % 4 == 0 to avoid early update.

3. Final Answer:

   optimizer.step() and zero_grad() are called too early at i=0 -> Option A

4. Quick Check:

   Modulo condition triggers step too early [OK]

Hint: Use (i + 1) % N == 0 to delay step until after N batches [OK]

Common Mistakes:

• Calling optimizer.step() at first batch
• Confusing zero_grad() timing
• Ignoring off-by-one in modulo condition

5. You want to train a large model on a GPU with limited memory using gradient accumulation with batch size 64 split into 8 steps. Which approach correctly adjusts the learning rate and loss scaling to keep training stable?

hard

A. Divide the loss by 8 before backward and keep the original learning rate

B. Multiply the learning rate by 8 and do not scale the loss

C. Divide the learning rate by 8 and multiply the loss by 8

D. Keep loss and learning rate unchanged, just accumulate gradients

Solution

1. Step 1: Understand gradient accumulation effect on loss

   Splitting batch into 8 steps means gradients accumulate over 8 mini-batches, so loss must be divided by 8 to average gradients correctly.

2. Step 2: Adjust learning rate accordingly

   Learning rate stays the same because gradient accumulation simulates a larger batch size without changing optimization dynamics.

3. Final Answer:

   Divide the loss by 8 before backward and keep the original learning rate -> Option A

4. Quick Check:

   Loss scaled down, learning rate unchanged for stable accumulation [OK]

Hint: Scale loss down by steps, keep learning rate same [OK]

Common Mistakes:

• Scaling learning rate instead of loss
• Not scaling loss causing gradient explosion
• Assuming no changes needed for accumulation