Gradient accumulation and zeroing in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of gradient accumulation with zeroing

Consider the following PyTorch code snippet that performs gradient accumulation over two mini-batches before updating the model parameters. What will be the value of model.linear.weight.grad after the second backward call and before optimizer.step()?

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1, bias=False)

model = SimpleModel()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Input and target batches
inputs1 = torch.tensor([[1.0]])
targets1 = torch.tensor([[2.0]])
inputs2 = torch.tensor([[2.0]])
targets2 = torch.tensor([[4.0]])

criterion = nn.MSELoss()

# First mini-batch
outputs1 = model(inputs1)
loss1 = criterion(outputs1, targets1)
loss1.backward()  # Gradients computed and accumulated

# Second mini-batch
outputs2 = model(inputs2)
loss2 = criterion(outputs2, targets2)
loss2.backward()  # Gradients accumulated again

# What is model.linear.weight.grad here?

A A tensor approximately equal to [[-15.0]] representing triple the gradient from one batch

B A tensor approximately equal to [[-10.0]] representing the sum of gradients from both batches

C None, because gradients are zeroed automatically after each backward call

D A tensor approximately equal to [[-5.0]] representing only the gradient from the second batch

💡 Hint

Remember that calling backward accumulates gradients unless you zero them explicitly.

In PyTorch, gradients accumulate by default. After the first backward call, the gradient is computed for the first batch. The second backward call adds the gradient from the second batch to the existing gradient. Since the model is a simple linear layer and the loss is MSE, the gradients add up. The value is approximately the sum of gradients from both batches.

🧠 Conceptual

intermediate

Why zero gradients before backward?

Why do we usually call optimizer.zero_grad() before calling loss.backward() in a training loop?

A To clear old gradients so that new gradients are not accumulated on top of them

B To reset model weights to zero before each update

C To initialize the optimizer's learning rate

D To save memory by deleting the computation graph

💡 Hint

Think about what happens if you don't clear gradients before backward.

Calling optimizer.zero_grad() clears the gradients from the previous backward pass. Without this, gradients would accumulate, causing incorrect updates. This is important to ensure each update step uses only the current batch's gradients.

❓ Hyperparameter

advanced

Choosing accumulation steps for gradient accumulation

You want to simulate a batch size of 64 but your GPU can only handle batch size 16. You decide to use gradient accumulation with accumulation_steps. Which value of accumulation_steps correctly simulates the larger batch size?

A 8, because 16 / 8 = 2

B 2, because 16 + 2 = 18

C 4, because 16 * 4 = 64

D 1, because no accumulation is needed

💡 Hint

Think about how many small batches you need to accumulate to reach the target batch size.

Gradient accumulation steps multiply the effective batch size. If your GPU can handle 16 samples at once, accumulating gradients over 4 steps simulates a batch size of 64 (16 * 4).

🔧 Debug

advanced

Why does the model not train when zero_grad is missing?

You wrote this training loop but the model's loss does not decrease over epochs. What is the most likely cause?

PyTorch

for epoch in range(5):
    for inputs, targets in dataloader:
        outputs = model(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()
        # Missing optimizer.zero_grad()

A The optimizer.step() is called before loss.backward(), causing errors

B The model weights are reset to zero every batch, so no learning happens

C The loss.backward() is not called, so no gradients are computed

D Gradients accumulate every batch causing optimizer.step() to apply huge updates, destabilizing training

💡 Hint

What happens if gradients are never cleared?

Without zeroing gradients, they accumulate every batch. This causes optimizer.step() to apply very large updates, which can prevent the model from converging and cause loss to stay high or fluctuate.

❓ Metrics

expert

Effect of gradient accumulation on training metrics

You train a model with gradient accumulation over 4 steps and batch size 16, simulating batch size 64. Which of the following statements about training loss and accuracy metrics logged per optimizer step is true?

A Loss and accuracy logged per optimizer step correspond to the combined effect of 64 samples, so they are smoother and more stable

B Loss and accuracy logged per optimizer step correspond to only 16 samples, so they are noisier

C Loss and accuracy do not change because accumulation does not affect metrics

D Loss and accuracy are invalid because gradients are accumulated

💡 Hint

Think about what batch size the optimizer step represents when using accumulation.

When accumulating gradients over 4 steps of batch size 16, the optimizer step updates parameters as if using batch size 64. Metrics logged per optimizer step reflect this larger effective batch size, resulting in smoother and more stable values.

Practice

1. What is the main purpose of zeroing gradients in PyTorch during training?

easy

A. To clear old gradients before computing new ones

B. To increase the learning rate automatically

C. To save the model weights to disk

D. To initialize the model parameters

Solution

1. Step 1: Understand gradient accumulation process

   Gradients accumulate by default in PyTorch, so old gradients must be cleared before new backward passes.

2. Step 2: Purpose of zeroing gradients

   Zeroing gradients resets them to zero to avoid mixing old and new gradient information.

3. Final Answer:

   To clear old gradients before computing new ones -> Option A

4. Quick Check:

   Zeroing gradients = clear old gradients [OK]

Hint: Zero gradients before backward to avoid mixing old info [OK]

Common Mistakes:

• Thinking zeroing resets model weights
• Confusing zeroing with saving model
• Believing zeroing changes learning rate

2. Which of the following is the correct way to zero gradients of a model named model in PyTorch?

easy

A. model.zero_grad()

B. model.clear_grad()

C. model.reset_gradients()

D. model.zero_gradients()

Solution

1. Step 1: Recall PyTorch API for zeroing gradients

   The official method to zero gradients is zero_grad() on the model or optimizer.

2. Step 2: Check method names

   Only model.zero_grad() is a valid PyTorch method; others are incorrect or do not exist.

3. Final Answer:

   model.zero_grad() -> Option A

4. Quick Check:

   Correct method name = zero_grad() [OK]

Hint: Use model.zero_grad() to clear gradients [OK]

Common Mistakes:

• Using non-existent methods like clear_grad()
• Confusing zero_grad with reset_gradients
• Adding extra letters to method names

3. Consider this PyTorch code snippet:

optimizer.zero_grad()
for i, batch in enumerate(data_loader):
    outputs = model(batch)
    loss = loss_fn(outputs, targets)
    loss.backward()
    if (i + 1) % 4 == 0:
        optimizer.step()
        optimizer.zero_grad()

What is the effect of this code on gradient updates?

medium

A. Updates model every batch without accumulation

B. Updates model only once after all batches

C. Does not update model at all

D. Accumulates gradients over 4 batches before updating

Solution

1. Step 1: Analyze the loop and optimizer calls

   Gradients are zeroed before loop and after every 4 batches. Backward is called every batch, so gradients accumulate for 4 batches.

2. Step 2: Understand optimizer.step() timing

   Optimizer updates weights every 4 batches, using accumulated gradients, then zeroes gradients to start fresh.

3. Final Answer:

   Accumulates gradients over 4 batches before updating -> Option D

4. Quick Check:

   Step every 4 batches = gradient accumulation [OK]

Hint: Step optimizer after several backward calls to accumulate [OK]

Common Mistakes:

• Assuming update happens every batch
• Forgetting gradients accumulate without zeroing
• Thinking zero_grad inside loop clears gradients too early

4. Identify the error in this PyTorch training loop:

for batch in data_loader:
    optimizer.zero_grad()
    outputs = model(batch)
    loss = loss_fn(outputs, targets)
    loss.backward()
    optimizer.step()
    loss.backward()

What is wrong here?

medium

A. Not calling optimizer.zero_grad() before backward

B. Missing optimizer.step() call

C. Calling loss.backward() twice without zeroing gradients

D. Using loss.backward() before computing loss

Solution

1. Step 1: Review order of backward and optimizer calls

   Loss.backward() is called twice without zeroing gradients in between, causing gradient accumulation unintentionally.

2. Step 2: Understand consequences of double backward

   Second backward adds gradients again, which is usually a mistake unless intentional accumulation is desired.

3. Final Answer:

   Calling loss.backward() twice without zeroing gradients -> Option C

4. Quick Check:

   Double backward without zero_grad causes wrong gradients [OK]

Hint: Call backward once per batch unless accumulating intentionally [OK]

Common Mistakes:

• Calling backward multiple times without zeroing
• Forgetting zero_grad before backward
• Misplacing optimizer.step() call

5. You want to train a large model on a GPU with limited memory by accumulating gradients over 8 mini-batches before updating weights. Which sequence correctly implements this in PyTorch?

hard

A. Call optimizer.zero_grad() every batch, call loss.backward() every batch, call optimizer.step() every batch

B. Call optimizer.zero_grad() once before loop, call loss.backward() every batch, call optimizer.step() and zero_grad() every 8 batches

C. Call optimizer.zero_grad() every 8 batches, call loss.backward() every 8 batches, call optimizer.step() every batch

D. Call optimizer.zero_grad() once before loop, call loss.backward() every 8 batches, call optimizer.step() every batch

Solution

1. Step 1: Understand gradient accumulation logic

   Gradients must be zeroed once before accumulation starts, then loss.backward() called every batch to accumulate gradients.

2. Step 2: Timing optimizer step and zero_grad

   After accumulating gradients over 8 batches, call optimizer.step() to update weights, then zero_grad() to reset gradients for next accumulation.

3. Final Answer:

   Call optimizer.zero_grad() once before loop, call loss.backward() every batch, call optimizer.step() and zero_grad() every 8 batches -> Option B

4. Quick Check:

   Zero once, backward every batch, step and zero every 8 batches [OK]

Hint: Zero once, backward each batch, step and zero after N batches [OK]

Common Mistakes:

• Zeroing gradients every batch breaks accumulation
• Calling backward only every N batches misses gradients
• Stepping optimizer too often or too rarely