requires_grad flag in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a tensor that tracks gradients.

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=[1])

Drag options to blanks, or click blank then click option'

ANone

BFalse

CTrue

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Using False disables gradient tracking, so gradients won't be computed.

Using None or 0 are invalid values for requires_grad.

Setting requires_grad=True tells PyTorch to track operations on the tensor for gradient computation.

2fill in blank

medium

Complete the code to stop gradient tracking on a tensor.

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
x = x.detach()
x.requires_grad = [1]

Drag options to blanks, or click blank then click option'

ATrue

BNone

C0

DFalse

Attempts:

3 left

💡 Hint

Common Mistakes

Setting requires_grad to True keeps tracking gradients.

Using None or 0 are not valid boolean values for requires_grad.

Setting requires_grad = False disables gradient tracking for the tensor.

3fill in blank

hard

Fix the error in the code to enable gradient tracking on a new tensor copied from x.

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = x.clone()
y.requires_grad = [1]

Drag options to blanks, or click blank then click option'

ATrue

BNone

CFalse

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Leaving requires_grad as False means no gradients will be tracked.

Using None or 0 are invalid for requires_grad.

After cloning, set requires_grad=True to track gradients on the new tensor.

4fill in blank

hard

Fill both blanks to create a tensor with gradient tracking and then disable it.

PyTorch

import torch
x = torch.tensor([4.0, 5.0], requires_grad=[1])
x.requires_grad = [2]

Drag options to blanks, or click blank then click option'

ATrue

BFalse

CNone

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Setting requires_grad to None or 0 causes errors.

Not disabling gradient tracking when needed.

First, create the tensor with requires_grad=True to track gradients, then disable it by setting requires_grad=False.

5fill in blank

hard

Fill all three blanks to create a tensor, clone it with gradients, and then disable gradients on the clone.

PyTorch

import torch
x = torch.tensor([7.0, 8.0], requires_grad=[1])
y = x.clone()
y.requires_grad = [2]
z = y.detach()
z.requires_grad = [3]

Drag options to blanks, or click blank then click option'

ATrue

BFalse

CNone

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Not enabling gradients on the clone.

Not disabling gradients after detaching.

Create x with gradient tracking enabled (True), clone it to y which also tracks gradients (True), then detach z from the graph and disable gradients (False).

Practice

1. What does setting requires_grad=True on a PyTorch tensor do?

easy

A. It tells PyTorch to track operations on the tensor for gradient calculation.

B. It freezes the tensor so it won't change during training.

C. It converts the tensor to a NumPy array.

D. It deletes the tensor from memory.

Solution

1. Step 1: Understand the purpose of requires_grad

   The flag requires_grad=True tells PyTorch to remember all operations on the tensor so it can compute gradients later.

2. Step 2: Identify what happens when requires_grad is True

   This tracking is essential for training models because gradients are used to update weights.

3. Final Answer:

   It tells PyTorch to track operations on the tensor for gradient calculation. -> Option A

4. Quick Check:

   requires_grad=True means track gradients [OK]

Hint: True means track gradients for learning, False means no tracking [OK]

Common Mistakes:

• Thinking requires_grad freezes the tensor
• Confusing requires_grad with tensor conversion
• Assuming it deletes the tensor

2. Which of the following is the correct way to create a tensor with gradient tracking enabled?

easy

A. torch.tensor([1.0, 2.0, 3.0], grad=True)

B. torch.tensor([1.0, 2.0, 3.0], requires_grad=False)

C. torch.tensor([1.0, 2.0, 3.0], requires_grad=True)

D. torch.tensor([1.0, 2.0, 3.0], track_grad=True)

Solution

1. Step 1: Recall the correct argument name

   The correct argument to enable gradient tracking is requires_grad=True.

2. Step 2: Check each option

   torch.tensor([1.0, 2.0, 3.0], requires_grad=True) uses requires_grad=True, which is correct. Options C and D use invalid argument names, and B disables gradient tracking.

3. Final Answer:

   torch.tensor([1.0, 2.0, 3.0], requires_grad=True) -> Option C

4. Quick Check:

   Use requires_grad=True to track gradients [OK]

Hint: Use requires_grad=True exactly to enable gradients [OK]

Common Mistakes:

• Using incorrect argument names like grad or track_grad
• Setting requires_grad to False by mistake
• Forgetting to set requires_grad at tensor creation

3. What will be the output of this code snippet?

import torch
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x * 2
y.backward(torch.tensor([1.0, 1.0]))
print(x.grad)

medium

A. tensor([2., 2.])

B. tensor([1., 1.])

C. tensor([4., 6.])

D. None

Solution

1. Step 1: Understand the operations and gradient flow

   y = x * 2 means each element in x is multiplied by 2. The gradient of y with respect to x is 2 for each element.

2. Step 2: Apply backward with gradient vector

   Calling y.backward with tensor([1.0, 1.0]) sums gradients weighted by 1. So x.grad will be tensor([2., 2.]).

3. Final Answer:

   tensor([2., 2.]) -> Option A

4. Quick Check:

   Gradient of 2*x is 2 [OK]

Hint: Gradient of x*2 is 2, so x.grad = [2, 2] [OK]

Common Mistakes:

• Expecting gradient to be the original tensor values
• Confusing output with input tensor
• Forgetting to call backward()

4. Identify the error in this code snippet related to gradient tracking:

import torch
x = torch.tensor([1.0, 2.0, 3.0])
y = x * 3
y.backward()

medium

A. x was not created with requires_grad=True, so backward() will fail.

B. There is no error; the code runs fine.

C. The backward() call is missing a gradient argument.

D. The tensor y is not a scalar, so backward() cannot be called.

Solution

1. Step 1: Check requires_grad flag on x

   x was created without requires_grad=True, so it does not track gradients. This means y also does not track gradients, causing backward() to fail.

2. Step 2: Understand backward() requirements

   The error occurs because tensors need grad_fn (from requires_grad=True) to call backward(). Additionally, non-scalar tensors need a gradient argument, so calling backward() without it on a vector tensor causes an error.

3. Final Answer:

   The tensor y is not a scalar, so backward() cannot be called. -> Option D

4. Quick Check:

   Non-scalar tensors require gradient argument in backward() [OK]

Hint: Non-scalar tensors need gradient argument in backward() [OK]

Common Mistakes:

• Forgetting requires_grad=True on tensors before backward()
• Ignoring requires_grad flag importance
• Not providing gradient argument for vector tensors

5. You want to freeze the first layer of a neural network during training to save memory. Which code snippet correctly disables gradient tracking for that layer's parameters?

hard

A. model.layer1.grad = None

B. for param in model.layer1.parameters():
param.requires_grad = False

C. model.layer1.requires_grad = False

D. for param in model.layer1.parameters():
param.requires_grad = True

Solution

1. Step 1: Understand how to freeze layers

   Freezing means stopping gradient calculation for parameters by setting their requires_grad to False.

2. Step 2: Check correct way to set requires_grad

   Parameters are tensors inside the layer. We must set param.requires_grad = False for each parameter. for param in model.layer1.parameters(): param.requires_grad = False does this correctly.

3. Final Answer:

   for param in model.layer1.parameters(): param.requires_grad = False -> Option B

4. Quick Check:

   Set requires_grad=False on parameters to freeze [OK]

Hint: Freeze by setting requires_grad=False on each parameter [OK]

Common Mistakes:

• Trying to set requires_grad on the whole layer object
• Setting requires_grad=True instead of False
• Confusing grad attribute with requires_grad