Backward pass (loss.backward) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the backward pass in PyTorch?

The backward pass computes gradients of the loss with respect to model parameters. It helps the model learn by showing how to adjust weights to reduce errors.

beginner

What does the method loss.backward() do in PyTorch?

It calculates the gradients of the loss tensor with respect to all tensors that have requires_grad=True. These gradients are stored in the .grad attribute of each tensor.

intermediate

Why do we need to call optimizer.zero_grad() before loss.backward()?

Because PyTorch accumulates gradients by default, calling optimizer.zero_grad() clears old gradients. This prevents mixing gradients from multiple backward passes.

beginner

What happens if you forget to call loss.backward() during training?

No gradients will be computed, so the optimizer cannot update the model weights. The model will not learn or improve.

intermediate

How does PyTorch know which operations to track for gradient computation?

PyTorch builds a computation graph dynamically during the forward pass. It tracks operations on tensors with requires_grad=True to compute gradients during the backward pass.

What does loss.backward() compute?

A The optimizer step

B Gradients of loss with respect to model parameters

C The updated model weights

D The loss value itself

loss.backward() calculates gradients needed for updating model weights.

Before calling loss.backward(), why do we call optimizer.zero_grad()?

A To clear old gradients so they don't accumulate

B To compute the loss value

C To update the model weights

D To save the model

Gradients accumulate by default, so clearing them prevents mixing updates.

If a tensor has requires_grad=False, what happens during loss.backward()?

A Gradient is computed normally

B The tensor is updated automatically

C An error occurs

D No gradient is computed for that tensor

Only tensors with requires_grad=True get gradients computed.

What is stored in the .grad attribute after loss.backward()?

A The updated tensor value

B The loss value

C The gradient of the loss with respect to the tensor

D The optimizer state

.grad holds the computed gradients for each parameter.

What does PyTorch use to track operations for gradient calculation?

A A dynamic computation graph

B A static computation graph

C A database

D A configuration file

PyTorch builds a dynamic graph during the forward pass to track operations.

Explain in your own words what happens during the backward pass when you call loss.backward() in PyTorch.

Think about how PyTorch figures out how to change weights to reduce loss.

Why is it important to call optimizer.zero_grad() before loss.backward() in a training loop?

Consider what happens if gradients from previous steps mix with current ones.