Gradient access (.grad) in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Gradient access (.grad)

Which metric matters for Gradient Access (.grad) and WHY

When working with gradients in PyTorch, the key metric is the gradient values themselves. These values show how much each model parameter should change to reduce the error. Accessing .grad helps us understand if the model is learning properly. If gradients are zero or very small, the model might not learn well. If gradients are too large, the model might be unstable.

Confusion matrix or equivalent visualization

For gradient access, we don't use a confusion matrix. Instead, we look at the gradient tensor values. For example, after a backward pass, a parameter's gradient might look like this:

    tensor([[ 0.01, -0.02],
            [ 0.00,  0.03]])
    

This shows how much each parameter will update. Monitoring these values helps detect issues like vanishing or exploding gradients.

Precision vs Recall tradeoff analogy for gradients

Think of gradients like directions on a map. If the directions are too weak (small gradients), you might not move enough to reach your goal (slow learning). If directions are too strong (large gradients), you might overshoot or get lost (unstable learning). The tradeoff is to have gradients that are just right -- strong enough to learn but not too strong to cause problems.

What "good" vs "bad" gradient values look like

• Good: Gradients have moderate values, not zero, not extremely large. They change smoothly during training.
• Bad: Gradients are all zeros (no learning), or very large values (causing unstable updates), or NaN values (training breaks).

Common pitfalls when accessing gradients

• Forgetting to call optimizer.zero_grad() before backward() causes gradients to accumulate unexpectedly.
• Accessing .grad before backward() returns None because gradients are not computed yet.
• Not detaching tensors properly can cause memory leaks when accessing gradients.
• Ignoring gradient clipping can lead to exploding gradients and unstable training.

Self-check question

Your model's parameters have gradients that are all zeros after backward(). Is your model learning? Why or why not?

Answer: No, the model is not learning because zero gradients mean no updates will happen to the parameters. This could be due to a bug, such as missing loss calculation, or the model output not depending on the parameters.

Key Result

Gradient values accessed via .grad indicate if model parameters are updating correctly; zero or extreme values signal learning issues.

Practice

1. What does the .grad attribute of a PyTorch tensor represent?

easy

A. The shape of the tensor

B. The value of the tensor itself

C. The gradient of the tensor after backpropagation

D. The data type of the tensor

Solution

1. Step 1: Understand the role of .grad in PyTorch

   The .grad attribute stores the gradient of the tensor computed during backpropagation.

2. Step 2: Differentiate from other tensor properties

   The tensor's value, shape, and data type are stored in other attributes, not .grad.

3. Final Answer:

   The gradient of the tensor after backpropagation -> Option C

4. Quick Check:

   .grad = gradient after backward() [OK]

Hint: Remember: .grad holds gradients after backward() [OK]

Common Mistakes:

• Confusing .grad with the tensor's value
• Expecting .grad before calling backward()
• Mixing .grad with tensor shape or dtype

2. Which of the following code snippets correctly accesses the gradient of a tensor x after backpropagation?

easy

A. grad = x.grad

B. grad = x.gradient()

C. grad = x.get_grad()

D. grad = x.gradient

Solution

1. Step 1: Recall the correct attribute for gradients

   PyTorch stores gradients in the .grad attribute, accessed as x.grad.

2. Step 2: Identify incorrect syntax

   Methods like gradient() or get_grad() do not exist for tensors.

3. Final Answer:

   grad = x.grad -> Option A

4. Quick Check:

   Use .grad attribute, not methods [OK]

Hint: Use .grad attribute, not functions [OK]

Common Mistakes:

• Trying to call .grad as a function
• Using incorrect method names
• Accessing .gradient instead of .grad

3. Consider the following PyTorch code:

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
z = y + 4
z.backward()
print(x.grad)

What will be the output of print(x.grad)?

medium

A. None

B. tensor(12.)

C. tensor(6.)

D. tensor(8.)

Solution

1. Step 1: Compute the derivative of z with respect to x

   Given y = x^3 and z = y + 4, the derivative is dz/dx = dy/dx = 3x^2.

2. Step 2: Calculate the gradient at x=2.0

   Plugging in x=2, 3 * (2)^2 = 3 * 4 = 12. So, x.grad will be tensor(12.).

3. Final Answer:

   tensor(12.) -> Option B

4. Quick Check:

   Derivative 3x^2 at x=2 = 12 [OK]

Hint: Calculate derivative manually, then plug in x [OK]

Common Mistakes:

• Forgetting to call backward() before accessing .grad
• Calculating derivative incorrectly
• Confusing y and z derivatives

4. You run this code:

import torch
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print(x.grad)

But the output is None. Why?

medium

A. You forgot to call y.backward() to compute gradients

B. The tensor x does not require gradients

C. You cannot access gradients of scalar tensors

D. The tensor x is not a float

Solution

1. Step 1: Check if backward() was called

   Gradients are computed only after calling backward() on the output tensor.

2. Step 2: Confirm requires_grad=True and missing backward()

   Since x has requires_grad=True, the missing step is calling y.backward().

3. Final Answer:

   You forgot to call y.backward() to compute gradients -> Option A

4. Quick Check:

   Call backward() before accessing .grad [OK]

Hint: Always call backward() before checking .grad [OK]

Common Mistakes:

• Expecting .grad before backward()
• Confusing requires_grad flag
• Thinking .grad auto-updates without backward()

5. Given this code:

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()
y.backward()
print(x.grad)

What will be the output of print(x.grad)?

hard

A. None

B. tensor([1., 2., 3.])

C. tensor([1., 4., 9.])

D. tensor([2., 4., 6.])

Solution

1. Step 1: Understand the function and gradient

   The function is y = sum(x_i^2). The gradient of y with respect to x_i is dy/dx_i = 2x_i.

2. Step 2: Calculate gradients for each element

   For x = [1, 2, 3], gradients are [2*1, 2*2, 2*3] = [2, 4, 6].

3. Final Answer:

   tensor([2., 4., 6.]) -> Option D

4. Quick Check:

   Gradient of x^2 is 2x element-wise [OK]

Hint: Gradient of x² is 2x for each element [OK]

Common Mistakes:

• Confusing sum of squares with squares of sums
• Forgetting element-wise gradient
• Expecting None without backward()