Gradient access (.grad) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of accessing .grad after backward()?

Consider the following PyTorch code snippet. What will be the value of x.grad after running it?

PyTorch

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
loss = y
loss.backward()
print(x.grad.item())

A 6.0

B 8.0

C 12.0

D None

💡 Hint

Recall the derivative of x^3 is 3*x^2.

The gradient of y = x^3 with respect to x is 3*x^2. At x=2, this is 3*4=12. So x.grad will be 12.0 after backward().

❓ Model Choice

intermediate

Which tensor will have .grad populated after backward()?

Given the following tensors, which one will have its .grad attribute populated after calling loss.backward()?

PyTorch

import torch
x = torch.tensor(1.0, requires_grad=True)
y = torch.tensor(2.0)
z = x * y
loss = z
loss.backward()

A x

B y

C z

D loss

💡 Hint

Only tensors with requires_grad=True track gradients.

Only x has requires_grad=True, so only x.grad will be populated after backward().

❓ Hyperparameter

advanced

What happens if you call backward() twice without zeroing gradients?

In PyTorch, if you call loss.backward() twice on the same graph without zeroing gradients, what will happen to x.grad?

PyTorch

import torch
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
loss = y
loss.backward()
loss.backward()
print(x.grad.item())

A The gradient will be zero

B The gradient will be reset to the last backward call (9.0)

C An error will be raised

D The gradient will be doubled (12.0)

💡 Hint

Gradients accumulate by default in PyTorch.

Calling backward() twice accumulates gradients. The derivative of x^2 at x=3 is 2*x=6. After two backward calls, x.grad is 6 + 6 = 12.0.

🔧 Debug

advanced

Why is x.grad None after backward()?

Look at this code. Why is x.grad None after calling loss.backward()?

PyTorch

import torch
x = torch.tensor(4.0)
y = x ** 2
loss = y
loss.backward()
print(x.grad)

A x does not have requires_grad=True

B x.grad was manually set to None

C loss.backward() was not called

D loss is not a scalar

💡 Hint

Check if the tensor tracks gradients.

Since x was created without requires_grad=True, PyTorch does not track operations on it, so x.grad remains None.

🧠 Conceptual

expert

What is the shape and content of .grad for a tensor with multiple elements?

Given the code below, what will be the shape and values of x.grad after loss.backward()?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2
loss = y.sum()
loss.backward()
print(x.grad)

A Tensor(6.0) scalar

B Tensor([2.0, 4.0, 6.0]) with shape (3,)

C Tensor([1.0, 2.0, 3.0]) with shape (3,)

D None

💡 Hint

Derivative of x^2 is 2*x element-wise.

The gradient of sum(x^2) w.r.t x is 2*x element-wise. So x.grad is [2,4,6] with shape (3,).

Practice

1. What does the .grad attribute of a PyTorch tensor represent?

easy

A. The shape of the tensor

B. The value of the tensor itself

C. The gradient of the tensor after backpropagation

D. The data type of the tensor

Solution

1. Step 1: Understand the role of .grad in PyTorch

   The .grad attribute stores the gradient of the tensor computed during backpropagation.

2. Step 2: Differentiate from other tensor properties

   The tensor's value, shape, and data type are stored in other attributes, not .grad.

3. Final Answer:

   The gradient of the tensor after backpropagation -> Option C

4. Quick Check:

   .grad = gradient after backward() [OK]

Hint: Remember: .grad holds gradients after backward() [OK]

Common Mistakes:

• Confusing .grad with the tensor's value
• Expecting .grad before calling backward()
• Mixing .grad with tensor shape or dtype

2. Which of the following code snippets correctly accesses the gradient of a tensor x after backpropagation?

easy

A. grad = x.grad

B. grad = x.gradient()

C. grad = x.get_grad()

D. grad = x.gradient

Solution

1. Step 1: Recall the correct attribute for gradients

   PyTorch stores gradients in the .grad attribute, accessed as x.grad.

2. Step 2: Identify incorrect syntax

   Methods like gradient() or get_grad() do not exist for tensors.

3. Final Answer:

   grad = x.grad -> Option A

4. Quick Check:

   Use .grad attribute, not methods [OK]

Hint: Use .grad attribute, not functions [OK]

Common Mistakes:

• Trying to call .grad as a function
• Using incorrect method names
• Accessing .gradient instead of .grad

3. Consider the following PyTorch code:

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
z = y + 4
z.backward()
print(x.grad)

What will be the output of print(x.grad)?

medium

A. None

B. tensor(12.)

C. tensor(6.)

D. tensor(8.)

Solution

1. Step 1: Compute the derivative of z with respect to x

   Given y = x^3 and z = y + 4, the derivative is dz/dx = dy/dx = 3x^2.

2. Step 2: Calculate the gradient at x=2.0

   Plugging in x=2, 3 * (2)^2 = 3 * 4 = 12. So, x.grad will be tensor(12.).

3. Final Answer:

   tensor(12.) -> Option B

4. Quick Check:

   Derivative 3x^2 at x=2 = 12 [OK]

Hint: Calculate derivative manually, then plug in x [OK]

Common Mistakes:

• Forgetting to call backward() before accessing .grad
• Calculating derivative incorrectly
• Confusing y and z derivatives

4. You run this code:

import torch
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print(x.grad)

But the output is None. Why?

medium

A. You forgot to call y.backward() to compute gradients

B. The tensor x does not require gradients

C. You cannot access gradients of scalar tensors

D. The tensor x is not a float

Solution

1. Step 1: Check if backward() was called

   Gradients are computed only after calling backward() on the output tensor.

2. Step 2: Confirm requires_grad=True and missing backward()

   Since x has requires_grad=True, the missing step is calling y.backward().

3. Final Answer:

   You forgot to call y.backward() to compute gradients -> Option A

4. Quick Check:

   Call backward() before accessing .grad [OK]

Hint: Always call backward() before checking .grad [OK]

Common Mistakes:

• Expecting .grad before backward()
• Confusing requires_grad flag
• Thinking .grad auto-updates without backward()

5. Given this code:

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()
y.backward()
print(x.grad)

What will be the output of print(x.grad)?

hard

A. None

B. tensor([1., 2., 3.])

C. tensor([1., 4., 9.])

D. tensor([2., 4., 6.])

Solution

1. Step 1: Understand the function and gradient

   The function is y = sum(x_i^2). The gradient of y with respect to x_i is dy/dx_i = 2x_i.

2. Step 2: Calculate gradients for each element

   For x = [1, 2, 3], gradients are [2*1, 2*2, 2*3] = [2, 4, 6].

3. Final Answer:

   tensor([2., 4., 6.]) -> Option D

4. Quick Check:

   Gradient of x^2 is 2x element-wise [OK]

Hint: Gradient of x² is 2x for each element [OK]

Common Mistakes:

• Confusing sum of squares with squares of sums
• Forgetting element-wise gradient
• Expecting None without backward()