Zeroing gradients in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does zeroing gradients mean in PyTorch?

Zeroing gradients means setting all the gradients of model parameters to zero before starting a new backward pass. This prevents accumulation of gradients from multiple passes.

beginner

Why do we need to zero gradients before calling backward() in PyTorch?

Because PyTorch accumulates gradients by default, zeroing them ensures that gradients from previous batches do not mix with the current batch's gradients.

beginner

Which PyTorch method is commonly used to zero gradients?

The method optimizer.zero_grad() is commonly used to reset all gradients to zero before the backward pass.

intermediate

What happens if you forget to zero gradients in a training loop?

Gradients will accumulate across batches, causing incorrect updates and making training unstable or ineffective.

beginner

Show a simple PyTorch training loop snippet that includes zeroing gradients.

for data, target in dataloader: optimizer.zero_grad() # Reset gradients output = model(data) loss = loss_fn(output, target) loss.backward() # Compute gradients optimizer.step() # Update parameters

What is the purpose of calling optimizer.zero_grad() in PyTorch?

A To reset gradients to zero before computing new gradients

B To update model parameters

C To compute the loss value

D To save the model state

optimizer.zero_grad() clears old gradients so new gradients from the current batch do not accumulate with previous ones.

If you skip zeroing gradients, what will happen during training?

A Gradients will accumulate, causing incorrect updates

B Model parameters will not update

C Loss will always be zero

D Training will be faster

Without zeroing, gradients add up across batches, leading to wrong parameter updates.

Which PyTorch function computes gradients for model parameters?

A model.eval()

B optimizer.step()

C loss.backward()

D optimizer.zero_grad()

loss.backward() calculates gradients of the loss with respect to model parameters.

When should zeroing gradients be done in the training loop?

A After optimizer.step()

B Before loss.backward()

C After loss.backward()

D At the end of training

Zeroing gradients must happen before computing new gradients with loss.backward().

What does optimizer.step() do in PyTorch?

A Computes the loss

B Resets gradients to zero

C Loads data into the model

D Updates model parameters using current gradients

optimizer.step() applies the computed gradients to update model parameters.

Explain why zeroing gradients is important in a PyTorch training loop.

Think about what happens if gradients keep adding up over batches.

Describe the sequence of steps in a PyTorch training loop involving gradients.

Focus on the order of zeroing, backward, and stepping.