Gradient access (.grad) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the .grad attribute in PyTorch represent?

The .grad attribute holds the gradient of a tensor after backpropagation. It shows how much the tensor affects the final output.

beginner

When is the .grad attribute populated in PyTorch?

The .grad attribute is filled after calling backward() on a tensor that is part of a computation graph with requires_grad=True.

beginner

How do you enable gradient tracking for a tensor in PyTorch?

Set requires_grad=True when creating the tensor, for example: torch.tensor([1.0, 2.0], requires_grad=True).

beginner

What happens if you try to access .grad before calling backward()?

The .grad attribute will be None because gradients have not been computed yet.

intermediate

How can you clear gradients stored in .grad before the next backward pass?

Use optimizer.zero_grad() or manually set tensor.grad.zero_() to reset gradients to zero.

What must be set to True for a tensor to track gradients in PyTorch?

A requires_grad

B track_grad

C grad_enabled

D enable_grad

The requires_grad flag enables gradient tracking for tensors.

When is the .grad attribute of a tensor updated?

A After calling <code>backward()</code>

B When the tensor is created

C After calling <code>forward()</code>

D When printing the tensor

Gradients are computed and stored in .grad after backward() is called.

What is the value of .grad before any backward pass?

A Same as the tensor

B Zero tensor

C Random values

D None

Before backward, .grad is None because no gradients are computed yet.

How do you reset gradients in PyTorch before a new training step?

A Set <code>requires_grad=False</code>

B Call <code>optimizer.zero_grad()</code>

C Call <code>tensor.backward()</code>

D Delete the tensor

Calling optimizer.zero_grad() clears gradients to avoid accumulation.

Which of these tensors will have a .grad attribute after backward?

A Tensor with <code>requires_grad=False</code>

B Tensor created with <code>torch.no_grad()</code>

C Tensor with <code>requires_grad=True</code>

D Tensor converted to numpy

Only tensors with requires_grad=True track gradients and have .grad populated.

Explain how to access and use the .grad attribute in PyTorch during training.

Think about the steps in a training loop involving gradients.

Describe what happens if you forget to reset gradients before the next backward pass in PyTorch.

Consider how gradients behave across multiple backward calls.