Gradient accumulation and zeroing in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient accumulation in PyTorch?

Gradient accumulation is a technique where gradients are summed over multiple mini-batches before updating model weights. This helps simulate a larger batch size without increasing memory usage.

beginner

Why do we need to zero gradients in PyTorch during training?

We zero gradients to clear old gradient values from the previous backward pass. Without zeroing, gradients would keep accumulating unintentionally, leading to incorrect updates.

beginner

How do you zero gradients in PyTorch?

You call optimizer.zero_grad() before the backward pass to reset all gradients to zero.

intermediate

What happens if you forget to zero gradients in a training loop?

Gradients from all previous batches accumulate, causing the model to update weights incorrectly and potentially harming training performance.

intermediate

How does gradient accumulation help when GPU memory is limited?

By accumulating gradients over several small batches, you can simulate a larger batch size without needing to load all data at once, saving memory.

What PyTorch function is used to clear gradients before a backward pass?

A optimizer.step()

B model.zero_grad()

C loss.backward()

D optimizer.zero_grad()

optimizer.zero_grad() resets all gradients to zero before computing new gradients.

Why accumulate gradients over multiple batches?

A To increase effective batch size without extra memory

B To speed up training by skipping backward passes

C To avoid zeroing gradients

D To reduce model size

Accumulating gradients simulates a larger batch size without needing more memory.

What happens if you call optimizer.step() without zeroing gradients first?

A Weights update with accumulated gradients from previous steps

B Weights do not update

C Training crashes

D Gradients reset automatically

Gradients accumulate if not zeroed, so weights update with combined gradients.

Which of these is NOT a reason to use gradient accumulation?

A Limited GPU memory

B Simulate larger batch size

C Avoid zeroing gradients

D Improve training stability

Gradient accumulation requires careful zeroing; it does not avoid zeroing gradients.

When should optimizer.zero_grad() be called in the training loop?

A After optimizer.step()

B Before loss.backward()

C After loss.backward()

D At the end of training

Zeroing gradients before backward pass ensures gradients are fresh for current batch.

Explain how gradient accumulation works and why it is useful in training deep learning models.

Think about training with small batches but wanting the effect of a big batch.

Describe the importance of zeroing gradients in PyTorch and what could happen if you skip this step.

Consider what happens if gradients keep adding up every batch.