Zeroing gradients in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to zero the gradients of the optimizer before the backward pass.

PyTorch

optimizer.[1]()

Drag options to blanks, or click blank then click option'

Azero_grad

Bstep

Cbackward

Dzero

Attempts:

3 left

💡 Hint

Common Mistakes

Using optimizer.step() instead of zero_grad() to clear gradients.

Calling backward() on the optimizer which is incorrect.

In PyTorch, optimizer.zero_grad() clears old gradients before computing new ones.

2fill in blank

medium

Complete the code to zero the gradients of the model's parameters before backpropagation.

PyTorch

for param in model.parameters():
    if param.grad is not None:
        param.grad.[1]()

Drag options to blanks, or click blank then click option'

Azero_grad

Bzero_

Cclear

Dreset

Attempts:

3 left

💡 Hint

Common Mistakes

Using zero_grad() on the gradient tensor which does not exist.

Using clear() or reset() which are not valid tensor methods.

Each parameter's gradient tensor can be zeroed using param.grad.zero_() in PyTorch.

3fill in blank

hard

Fix the error in zeroing gradients before the backward pass.

PyTorch

optimizer.[1]()
loss.backward()
optimizer.step()

Drag options to blanks, or click blank then click option'

Azero_grad

Bstep

Cbackward

Dclear_grad

Attempts:

3 left

💡 Hint

Common Mistakes

Calling optimizer.step() before zeroing gradients.

Using a non-existent method like clear_grad().

Before calling loss.backward(), gradients must be cleared with optimizer.zero_grad().

4fill in blank

hard

Fill both blanks to zero gradients for all parameters in the model.

PyTorch

for param in model.parameters():
    if param.grad is not None:
        param.grad.[1]()
model.[2]()

Drag options to blanks, or click blank then click option'

Azero_

Bstep

Czero_grad

Dbackward

Attempts:

3 left

💡 Hint

Common Mistakes

Using step() instead of zero_grad() to clear gradients.

Confusing backward() with zeroing gradients.

Zero gradients of each parameter with param.grad.zero_() and zero model gradients with model.zero_grad().

5fill in blank

hard

Fill all three blanks to zero gradients, perform backward pass, and update weights.

PyTorch

optimizer.[1]()
loss.[2]()
optimizer.[3]()

Drag options to blanks, or click blank then click option'

Astep

Bbackward

Czero_grad

Dclear

Attempts:

3 left

💡 Hint

Common Mistakes

Calling step() before zero_grad() or backward().

Using clear() which is not a valid optimizer method.

First zero gradients with optimizer.zero_grad(), then compute gradients with loss.backward(), finally update weights with optimizer.step().

Practice

1. Why do we need to zero gradients in PyTorch before calling loss.backward() during training?

easy

A. To initialize the model parameters randomly

B. To save memory by deleting the model weights

C. To speed up the training by skipping backward pass

D. To clear old gradients so new gradients are calculated correctly

Solution

1. Step 1: Understand gradient accumulation in PyTorch

   PyTorch accumulates gradients by default, so old gradients stay unless cleared.

2. Step 2: Purpose of zeroing gradients before backward pass

   Zeroing gradients ensures only the current batch's gradients are used for updating weights.

3. Final Answer:

   To clear old gradients so new gradients are calculated correctly -> Option D

4. Quick Check:

   Zero gradients = fresh gradient calculation [OK]

Hint: Always zero gradients before backward pass to avoid accumulation [OK]

Common Mistakes:

• Skipping zero_grad() causes wrong gradient sums
• Confusing zeroing gradients with resetting weights
• Thinking zero_grad() speeds training by skipping steps

2. Which of the following is the correct way to zero gradients for an optimizer named optimizer in PyTorch?

easy

A. optimizer.clear_grad()

B. optimizer.zero_gradients()

C. optimizer.zero_grad()

D. optimizer.reset_grad()

Solution

1. Step 1: Recall PyTorch optimizer API

   The correct method to zero gradients is zero_grad() on the optimizer object.

2. Step 2: Check method names for zeroing gradients

   Only zero_grad() is valid; others are incorrect or do not exist.

3. Final Answer:

   optimizer.zero_grad() -> Option C

4. Quick Check:

   Use zero_grad() method to clear gradients [OK]

Hint: Remember method name ends with '_grad', not '_gradients' [OK]

Common Mistakes:

• Using zero_gradients() which does not exist
• Confusing with clear_grad() or reset_grad() methods
• Calling zero_grad on model instead of optimizer

3. What will be the output of the following PyTorch code snippet?

import torch
x = torch.tensor([2.0], requires_grad=True)
optimizer = torch.optim.SGD([x], lr=0.1)
optimizer.zero_grad()
y = x * 3
loss = y ** 2
loss.backward()
print(x.grad.item())

medium

A. 36.0

B. 12.0

C. 18.0

D. 6.0

Solution

1. Step 1: Calculate loss and gradients

   y = 3 * x = 6.0; loss = y^2 = 36.0; gradient of loss wrt x is d(loss)/dx = 2*y * dy/dx = 2*6*3 = 36.

2. Step 2: Check printed gradient value

   x.grad will hold 36.0 after backward, but since x.grad is printed directly, it shows 36.0.

3. Final Answer:

   36.0 -> Option A

4. Quick Check:

   Gradient = 2*y*3 = 36.0 [OK]

Hint: Use chain rule: d(loss)/dx = 2*y * dy/dx [OK]

Common Mistakes:

• Calculating gradient as 12.0 by missing chain rule
• Confusing loss value with gradient value
• Forgetting to call zero_grad() before backward

4. Consider this training loop snippet:

for data, target in dataloader:
    output = model(data)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

What is the problem here?

medium

A. No problem, this is the correct order

B. Gradients are zeroed after optimizer.step(), causing stale gradients in next iteration

C. loss.backward() should be called after optimizer.step()

D. optimizer.zero_grad() should be called before loss.backward() to clear old gradients

Solution

1. Step 1: Understand correct order of zeroing gradients

   Gradients must be zeroed before calling loss.backward() to avoid accumulation.

2. Step 2: Identify error in given code

   Here, zero_grad() is called after optimizer.step(), so old gradients accumulate during backward pass.

3. Final Answer:

   Gradients are zeroed after optimizer.step(), causing stale gradients in next iteration -> Option B

4. Quick Check:

   Zero gradients before backward pass [OK]

Hint: Call zero_grad() before backward() to reset gradients [OK]

Common Mistakes:

• Calling zero_grad() after optimizer.step() instead of before backward()
• Swapping order of backward() and step()
• Assuming zero_grad() is optional

5. You want to accumulate gradients over 4 mini-batches before updating model weights to simulate a larger batch size. Which code snippet correctly implements this using zeroing gradients in PyTorch?

hard

A. optimizer.zero_grad()
for i, (data, target) in enumerate(dataloader):
output = model(data)
loss = loss_fn(output, target) / 4
loss.backward()
if (i + 1) % 4 == 0:
optimizer.step()
optimizer.zero_grad()

B. for i, (data, target) in enumerate(dataloader):
optimizer.zero_grad()
output = model(data)
loss = loss_fn(output, target)
loss.backward()
if (i + 1) % 4 == 0:
optimizer.step()

C. for i, (data, target) in enumerate(dataloader):
output = model(data)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
optimizer.zero_grad()

D. optimizer.zero_grad()
for data, target in dataloader:
output = model(data)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
optimizer.zero_grad()

Solution

1. Step 1: Understand gradient accumulation

   To accumulate gradients over multiple batches, zero_grad() is called once before loop, then backward() accumulates gradients.

2. Step 2: Check correct update and zeroing frequency

   After 4 batches, optimizer.step() updates weights, then zero_grad() clears gradients for next accumulation cycle.

3. Final Answer:

   Code snippet C correctly accumulates gradients and updates every 4 batches -> Option A

4. Quick Check:

   Zero once, accumulate 4 backward calls, then step and zero [OK]

Hint: Zero gradients once, step and zero every N batches to accumulate [OK]

Common Mistakes:

• Zeroing gradients inside loop every batch prevents accumulation
• Not dividing loss by accumulation steps causing large gradients
• Calling optimizer.step() every batch defeats accumulation purpose