Gradient accumulation in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient accumulation in training neural networks?

Gradient accumulation is a technique where gradients from multiple mini-batches are added together before updating the model weights. This simulates a larger batch size without needing more memory.

beginner

Why use gradient accumulation instead of increasing batch size directly?

Because hardware memory limits may prevent using large batches. Gradient accumulation lets you use small batches but still get the effect of a large batch by summing gradients over steps.

intermediate

In PyTorch, what must you do before calling loss.backward() when using gradient accumulation?

You should NOT call optimizer.zero_grad() every mini-batch. Instead, zero gradients only after the accumulated steps to keep adding gradients.

intermediate

How do you update model weights when using gradient accumulation with an accumulation step of 4?

You call optimizer.step() and optimizer.zero_grad() only every 4 mini-batches after accumulating gradients from each batch.

intermediate

What is a simple PyTorch code pattern for gradient accumulation?

For each mini-batch: compute loss, call loss.backward() without zeroing grads, then every N batches call optimizer.step() and optimizer.zero_grad() to update weights and reset gradients.

What does gradient accumulation help with?

A Increasing learning rate automatically

B Reducing the number of model parameters

C Simulating larger batch sizes without extra memory

D Stopping training early

Gradient accumulation sums gradients over multiple batches to simulate a larger batch size without needing more memory.

When using gradient accumulation, when should you call optimizer.zero_grad()?

A After accumulating gradients over several mini-batches

B After every mini-batch

C Before every mini-batch

D Never

You zero gradients only after accumulating over the desired number of mini-batches to keep adding gradients.

If your accumulation step is 3, how often do you call optimizer.step()?

A Every mini-batch

B Every 3 mini-batches

C Every 2 mini-batches

D Only once at the end

You update weights every 3 mini-batches after accumulating gradients from each.

Which of these is NOT a benefit of gradient accumulation?

A Improves model accuracy by default

B Allows training with large effective batch sizes on limited memory

C Helps stabilize training by simulating bigger batches

D Reduces memory usage compared to large batch training

Gradient accumulation helps with batch size and memory but does not automatically improve accuracy.

What happens if you forget to call optimizer.zero_grad() after accumulation?

A Training stops immediately

B Nothing, training continues normally

C Model weights reset to initial values

D Gradients keep accumulating, possibly causing incorrect updates

Not zeroing gradients causes them to accumulate indefinitely, leading to wrong updates.

Explain how gradient accumulation works and why it is useful in training neural networks.

Think about how small batches can add up to a big batch effect.

Describe the changes needed in a PyTorch training loop to implement gradient accumulation.

Focus on when to zero gradients and when to update weights.