Gradient accumulation in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of gradient accumulation with batch size 2

Consider the following PyTorch code snippet that uses gradient accumulation with accumulation_steps=2. What will be the printed value of model.linear.weight.grad after the loop?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1, bias=False)

model = SimpleModel()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = [torch.tensor([[1.0]]), torch.tensor([[2.0]])]
targets = [torch.tensor([[2.0]]), torch.tensor([[4.0]])]

accumulation_steps = 2
optimizer.zero_grad()

for i in range(2):
    output = model.linear(inputs[i])
    loss = (output - targets[i]).pow(2).mean()
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

print(model.linear.weight.grad)

A tensor([[1.]])

B None

C tensor([[0.]])

D tensor([[0.5]])

💡 Hint

Remember that after optimizer.step() and optimizer.zero_grad(), gradients are reset.

After the loop, optimizer.step() and optimizer.zero_grad() are called, so gradients are cleared. Therefore, model.linear.weight.grad is a tensor of zeros.

🧠 Conceptual

intermediate

Why use gradient accumulation?

Why do we use gradient accumulation in training deep learning models?

A To avoid using an optimizer during training.

B To reduce the number of model parameters by accumulating gradients.

C To speed up training by skipping backward passes.

D To simulate a larger batch size than fits in memory by accumulating gradients over smaller batches before updating weights.

💡 Hint

Think about memory limits and batch sizes.

Gradient accumulation allows training with an effective larger batch size by summing gradients over multiple smaller batches before updating model weights, which helps when GPU memory is limited.

❓ Hyperparameter

advanced

Effect of accumulation steps on learning rate

If you increase the number of gradient accumulation steps from 1 to 4 without changing the learning rate, what is the effective change in the learning rate per sample?

A The effective learning rate per sample is 4 times smaller.

B The effective learning rate per sample is 4 times larger.

C The effective learning rate per sample stays the same.

D The effective learning rate per sample becomes zero.

💡 Hint

Think about how many updates happen per sample when accumulating gradients.

With 4 accumulation steps, the model updates weights once every 4 batches, so each sample's gradient contributes less frequently, effectively reducing the learning rate per sample by 4 times.

🔧 Debug

advanced

Bug in gradient accumulation code

What error will the following PyTorch code raise?

import torch
import torch.nn as nn

model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

accumulation_steps = 2
optimizer.zero_grad()

for i in range(3):
    x = torch.tensor([[float(i + 1)]])
    y = torch.tensor([[2.0 * (i + 1)]])
    output = model(x)
    loss = (output - y).pow(2).mean()
    loss.backward()
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

optimizer.step()

A RuntimeError: optimizer.step() called twice without backward()

B RuntimeError: optimizer.step() called without zero_grad()

C RuntimeError: Trying to backward through the graph a second time

D No error, code runs fine

💡 Hint

Check how many times optimizer.step() is called relative to loss.backward().

The last optimizer.step() is called after the loop without a preceding loss.backward(), causing a RuntimeError because optimizer.step() expects gradients from a backward pass.

❓ Model Choice

expert

Choosing batch size and accumulation steps for limited GPU memory

You want to train a large transformer model but your GPU memory only fits batch size 8. You want an effective batch size of 32. Which approach is best?

A Use batch size 4 with accumulation steps = 8 to simulate batch size 32.

B Use batch size 8 with gradient accumulation steps = 4 to simulate batch size 32.

C Use batch size 8 and update weights after every batch without accumulation.

D Use batch size 32 and reduce model size to fit GPU memory.

💡 Hint

Think about memory limits and how accumulation simulates larger batch sizes.

Using batch size 8 with accumulation steps 4 allows simulating batch size 32 without exceeding GPU memory. Batch size 32 may not fit, and batch size 4 with accumulation 8 is less efficient due to smaller batches.