Backward pass (loss.backward) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of backward pass gradient accumulation

Consider the following PyTorch code snippet. What will be the value of x.grad after running all lines?

PyTorch

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x * 3
loss = y ** 2
loss.backward()
loss = (x * 4) ** 2
loss.backward()
print(x.grad.item())

A 100.0

B None (raises error)

C 16.0

D 36.0

💡 Hint

Remember that gradients accumulate by default in PyTorch unless you clear them.

The first backward computes gradient 36, the second computes 64, total 100 accumulated in x.grad.

🧠 Conceptual

intermediate

Effect of calling loss.backward() multiple times without zeroing gradients

In PyTorch, what happens if you call loss.backward() multiple times on different losses without resetting gradients?

A An error is raised on the second backward call

B Gradients accumulate, adding up from each backward call

C Gradients are overwritten by the latest backward call

D Gradients are reset to zero automatically before each backward

💡 Hint

Think about how PyTorch handles gradients by default.

PyTorch accumulates gradients in the .grad attribute by default. You must manually zero gradients to avoid accumulation.

❓ Hyperparameter

advanced

Choosing correct learning rate with backward pass

You notice your model's loss does not decrease after several backward passes and optimizer steps. Which learning rate adjustment is most likely to help?

A Remove the backward pass and only do forward passes

B Increase the learning rate to a larger value

C Keep the learning rate the same and increase batch size

D Decrease the learning rate to a smaller value

💡 Hint

Think about what happens if the learning rate is too high.

A too high learning rate can cause the model to overshoot minima and not converge. Decreasing it often helps training.

🔧 Debug

advanced

Identifying error in backward pass usage

What error will this PyTorch code raise when calling loss.backward()?

import torch
x = torch.tensor(1.0)
y = x * 2
loss = y ** 2
loss.backward()

A TypeError: 'float' object is not callable

B SyntaxError: invalid syntax

C RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

D No error, backward runs successfully

💡 Hint

Check if the tensor requires gradients.

The tensor x does not have requires_grad=True, so loss has no gradient function. Calling backward raises RuntimeError.

❓ Model Choice

expert

Choosing model for efficient backward pass in large-scale training

You want to train a very deep neural network with millions of parameters efficiently. Which approach helps reduce memory usage during the backward pass?

A Use gradient checkpointing to save memory by recomputing activations

B Use a shallow model to avoid backward pass complexity

C Disable gradient computation entirely with torch.no_grad()

D Increase batch size to reduce number of backward passes

💡 Hint

Think about trading computation for memory during backward.

Gradient checkpointing saves memory by not storing all intermediate activations, recomputing them during backward pass.