requires_grad flag in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of requires_grad flag after tensor operations?

Consider the following PyTorch code snippet. What will be the value of z.requires_grad after the operations?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
y = y.detach()
z = y + 1

A None

B True

C Raises an error

D False

💡 Hint

Think about what detach() does to the computation graph and the requires_grad flag.

The detach() method creates a new tensor that shares data but does not require gradients. Therefore, y.requires_grad becomes False, and so does z.requires_grad because it is computed from y.

❓ Model Choice

intermediate

Which tensor requires gradients for training a model?

You want to train a simple linear model in PyTorch. Which tensor should have requires_grad=True to update its values during training?

A Input data tensor

B Model parameters tensor

C Target labels tensor

D Random noise tensor

💡 Hint

Which tensors need gradients to update weights during backpropagation?

Model parameters need requires_grad=True so gradients are computed and weights updated. Input data and target labels do not require gradients.

❓ Hyperparameter

advanced

How does setting requires_grad=False affect training speed?

What is the effect of setting requires_grad=False on some model parameters during training?

A Causes training to fail with an error

B Slows down training because gradients are still computed

C Speeds up training by skipping gradient computation for those parameters

D Has no effect on training speed

💡 Hint

Think about what happens when gradients are not computed for some tensors.

Setting requires_grad=False tells PyTorch not to compute gradients for those tensors, reducing computation and speeding up training.

🔧 Debug

advanced

Why does this code raise an error related to requires_grad?

Examine the code below. Why does the backward call raise an error?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = x * 2
z = y.sum()
z.backward()

A Because x does not have requires_grad=True

B Because y is not a tensor

C Because sum() cannot be used before backward()

D Because backward() requires an argument

💡 Hint

Check if the tensor you want gradients for has requires_grad=True.

The tensor x was created without requires_grad=True, so PyTorch does not track operations for gradients. Calling backward() on z raises an error.

🧠 Conceptual

expert

What happens to requires_grad flag after in-place operations?

In PyTorch, if you perform an in-place operation on a tensor with requires_grad=True, what is the effect on the requires_grad flag and the computation graph?

A The <code>requires_grad</code> flag remains True, but the computation graph may be corrupted causing errors during backward

B The <code>requires_grad</code> flag automatically switches to False

C The tensor becomes detached from the graph but keeps <code>requires_grad=True</code>

D In-place operations are not allowed on tensors with <code>requires_grad=True</code> and raise an immediate error

💡 Hint

Consider how in-place changes affect gradient tracking and graph integrity.

In-place operations keep requires_grad=True but can corrupt the computation graph, leading to runtime errors during backpropagation. PyTorch warns about this but does not automatically disable gradient tracking.

Practice

1. What does setting requires_grad=True on a PyTorch tensor do?

easy

A. It tells PyTorch to track operations on the tensor for gradient calculation.

B. It freezes the tensor so it won't change during training.

C. It converts the tensor to a NumPy array.

D. It deletes the tensor from memory.

Solution

1. Step 1: Understand the purpose of requires_grad

   The flag requires_grad=True tells PyTorch to remember all operations on the tensor so it can compute gradients later.

2. Step 2: Identify what happens when requires_grad is True

   This tracking is essential for training models because gradients are used to update weights.

3. Final Answer:

   It tells PyTorch to track operations on the tensor for gradient calculation. -> Option A

4. Quick Check:

   requires_grad=True means track gradients [OK]

Hint: True means track gradients for learning, False means no tracking [OK]

Common Mistakes:

• Thinking requires_grad freezes the tensor
• Confusing requires_grad with tensor conversion
• Assuming it deletes the tensor

2. Which of the following is the correct way to create a tensor with gradient tracking enabled?

easy

A. torch.tensor([1.0, 2.0, 3.0], grad=True)

B. torch.tensor([1.0, 2.0, 3.0], requires_grad=False)

C. torch.tensor([1.0, 2.0, 3.0], requires_grad=True)

D. torch.tensor([1.0, 2.0, 3.0], track_grad=True)

Solution

1. Step 1: Recall the correct argument name

   The correct argument to enable gradient tracking is requires_grad=True.

2. Step 2: Check each option

   torch.tensor([1.0, 2.0, 3.0], requires_grad=True) uses requires_grad=True, which is correct. Options C and D use invalid argument names, and B disables gradient tracking.

3. Final Answer:

   torch.tensor([1.0, 2.0, 3.0], requires_grad=True) -> Option C

4. Quick Check:

   Use requires_grad=True to track gradients [OK]

Hint: Use requires_grad=True exactly to enable gradients [OK]

Common Mistakes:

• Using incorrect argument names like grad or track_grad
• Setting requires_grad to False by mistake
• Forgetting to set requires_grad at tensor creation

3. What will be the output of this code snippet?

import torch
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x * 2
y.backward(torch.tensor([1.0, 1.0]))
print(x.grad)

medium

A. tensor([2., 2.])

B. tensor([1., 1.])

C. tensor([4., 6.])

D. None

Solution

1. Step 1: Understand the operations and gradient flow

   y = x * 2 means each element in x is multiplied by 2. The gradient of y with respect to x is 2 for each element.

2. Step 2: Apply backward with gradient vector

   Calling y.backward with tensor([1.0, 1.0]) sums gradients weighted by 1. So x.grad will be tensor([2., 2.]).

3. Final Answer:

   tensor([2., 2.]) -> Option A

4. Quick Check:

   Gradient of 2*x is 2 [OK]

Hint: Gradient of x*2 is 2, so x.grad = [2, 2] [OK]

Common Mistakes:

• Expecting gradient to be the original tensor values
• Confusing output with input tensor
• Forgetting to call backward()

4. Identify the error in this code snippet related to gradient tracking:

import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = x * 3
y.backward()

medium

A. x was not created with requires_grad=True, so backward() will fail.

B. There is no error; the code runs fine.

C. The backward() call is missing a gradient argument.

D. The tensor y is not a scalar, so backward() cannot be called.

Solution

1. Step 1: Check requires_grad flag on x

   x was created without requires_grad=True, so it does not track gradients. This means y also does not track gradients, causing backward() to fail.

2. Step 2: Understand backward() requirements

   The error occurs because tensors need grad_fn (from requires_grad=True) to call backward(). Additionally, non-scalar tensors need a gradient argument, so calling backward() without it on a vector tensor causes an error.

3. Final Answer:

   The tensor y is not a scalar, so backward() cannot be called. -> Option D

4. Quick Check:

   Non-scalar tensors require gradient argument in backward() [OK]

Hint: Non-scalar tensors need gradient argument in backward() [OK]

Common Mistakes:

• Forgetting requires_grad=True on tensors before backward()
• Ignoring requires_grad flag importance
• Not providing gradient argument for vector tensors

5. You want to freeze the first layer of a neural network during training to save memory. Which code snippet correctly disables gradient tracking for that layer's parameters?

hard

A. model.layer1.grad = None

B. for param in model.layer1.parameters():
param.requires_grad = False

C. model.layer1.requires_grad = False

D. for param in model.layer1.parameters():
param.requires_grad = True

Solution

1. Step 1: Understand how to freeze layers

   Freezing means stopping gradient calculation for parameters by setting their requires_grad to False.

2. Step 2: Check correct way to set requires_grad

   Parameters are tensors inside the layer. We must set param.requires_grad = False for each parameter. for param in model.layer1.parameters(): param.requires_grad = False does this correctly.

3. Final Answer:

   for param in model.layer1.parameters(): param.requires_grad = False -> Option B

4. Quick Check:

   Set requires_grad=False on parameters to freeze [OK]

Hint: Freeze by setting requires_grad=False on each parameter [OK]

Common Mistakes:

• Trying to set requires_grad on the whole layer object
• Setting requires_grad=True instead of False
• Confusing grad attribute with requires_grad