Zeroing gradients in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this PyTorch code snippet?

Consider the following PyTorch code that performs a backward pass and then prints the gradient of a tensor. What will be printed?

PyTorch

import torch
x = torch.tensor([2.0], requires_grad=True)
y = x * 3
z = y ** 2
z.backward()
print(x.grad)
x.grad.zero_()
print(x.grad)

A tensor([36.])\ntensor([36.])

B tensor([36.])\ntensor([0.])

C tensor([6.])\ntensor([0.])

D tensor([6.])\ntensor([6.])

💡 Hint

Remember that backward() computes gradients and zero_() sets them to zero in-place.

The gradient of z = (3x)^2 = 9x^2 with respect to x is 18x. At x=2, gradient is 18*2=36. But since y = 3x and z = y^2, chain rule gives dz/dx = 2y * dy/dx = 2*(3x)*3 = 18x. So gradient is 36. However, the code prints x.grad after backward() and then after zeroing. The first print shows the gradient tensor, the second shows zero tensor after zero_().

❓ Model Choice

intermediate

Which PyTorch method correctly resets gradients before a new training step?

In a typical training loop, which method should be called to clear gradients before computing new ones?

A torch.no_grad()

B model.zero_grad()

C optimizer.zero_grad()

D loss.backward()

💡 Hint

Think about which object manages the parameters and their gradients during optimization.

The optimizer.zero_grad() method clears the gradients of all optimized tensors. This is necessary before calling loss.backward() to avoid gradient accumulation from previous steps.

❓ Hyperparameter

advanced

What happens if you forget to zero gradients in a PyTorch training loop?

Consider a training loop where optimizer.zero_grad() is never called. What is the effect on the gradients during training?

A Gradients accumulate, causing updates to be larger than intended.

B Gradients are reset automatically each step, so no effect.

C Training will raise a runtime error due to missing zeroing.

D Gradients become zero, so model weights do not update.

💡 Hint

Think about how PyTorch accumulates gradients by default.

PyTorch accumulates gradients on each backward call. If you don't zero them, gradients from previous steps add up, leading to larger updates than expected.

🔧 Debug

advanced

Why does this PyTorch code raise an error when zeroing gradients?

Examine the code below. Why does it raise an error at x.grad.zero_()?

PyTorch

import torch
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x.sum()
y.backward()
x.grad = None
x.grad.zero_()

A x.grad is None, so calling zero_() raises AttributeError.

B x.grad is not a tensor, so zero_() is undefined.

C x.grad has already been zeroed, so zero_() fails.

D x.grad is a scalar, zero_() requires a tensor.

💡 Hint

What happens if you assign None to a variable and then call a method on it?

Setting x.grad = None removes the gradient tensor. Then calling zero_() on None raises AttributeError because NoneType has no such method.

🧠 Conceptual

expert

Why is zeroing gradients important in mini-batch training?

In mini-batch gradient descent, why must gradients be zeroed before processing each batch?

A To save memory by deleting old gradients permanently.

B To initialize model weights to zero before each batch.

C Because gradients are automatically reset after optimizer.step().

D To prevent gradient accumulation from previous batches, ensuring correct updates.

💡 Hint

Consider how PyTorch handles gradients across multiple backward passes.

PyTorch accumulates gradients by default. Zeroing gradients before each batch ensures that only the current batch's gradients affect the update, preventing unintended accumulation.