Detaching from computation graph in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to detach the tensor from the computation graph.

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
z = y.sum()
detached = y.[1]()

Drag options to blanks, or click blank then click option'

Arequires_grad_

Bclone

Cdetach

Dbackward

Attempts:

3 left

💡 Hint

Common Mistakes

Using clone() does not detach from the graph.

Using backward() is for computing gradients, not detaching.

Using requires_grad_() changes gradient tracking but does not detach.

Using detach() creates a new tensor that shares data but is detached from the computation graph, so gradients won't flow back.

2fill in blank

medium

Complete the code to convert a detached tensor back to a NumPy array.

PyTorch

import torch
x = torch.tensor([4.0, 5.0, 6.0], requires_grad=True)
y = x * 3
detached = y.[1]()
numpy_array = detached.numpy()

Drag options to blanks, or click blank then click option'

Adetach

Bclone

Ccpu

Dnumpy

Attempts:

3 left

💡 Hint

Common Mistakes

Trying to convert a tensor with gradients directly to NumPy causes errors.

Using clone() instead of detach() does not stop gradient tracking.

First, detach the tensor to stop gradient tracking, then move it to CPU if needed, and finally convert to NumPy array. Here, only detach() is needed before numpy().

3fill in blank

hard

Fix the error in the code by detaching the tensor before converting to NumPy.

PyTorch

import torch
x = torch.tensor([7.0, 8.0, 9.0], requires_grad=True)
y = x * 4
numpy_array = y.[1]().numpy()

Drag options to blanks, or click blank then click option'

Adetach

Bclone

Ccpu

Drequires_grad_

Attempts:

3 left

💡 Hint

Common Mistakes

Calling numpy() directly on a tensor with requires_grad=True causes errors.

Using clone() does not detach the tensor.

Calling detach() removes the tensor from the computation graph, allowing safe conversion to NumPy.

4fill in blank

hard

Fill both blanks to detach a tensor and move it to CPU before converting to NumPy.

PyTorch

import torch
x = torch.tensor([10.0, 11.0, 12.0], requires_grad=True)
y = x * 5
result = y.[1]().[2]().numpy()

Drag options to blanks, or click blank then click option'

Adetach

Bcpu

Cclone

Dnumpy

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping the order causes errors.

Using clone() instead of detach() does not stop gradient tracking.

First detach the tensor to stop gradient tracking, then move it to CPU to ensure compatibility before converting to NumPy.

5fill in blank

hard

Fill all three blanks to detach a tensor, move it to CPU, and convert it to a NumPy array.

PyTorch

import torch
x = torch.tensor([13.0, 14.0, 15.0], requires_grad=True)
y = x * 6
numpy_array = y.[1]().[2]().[3]()

Drag options to blanks, or click blank then click option'

Adetach

Bcpu

Cnumpy

Dclone

Attempts:

3 left

💡 Hint

Common Mistakes

Calling numpy() before detach() causes errors.

Using clone() instead of detach() does not detach the tensor.

Detach the tensor to stop gradient tracking, move it to CPU, then convert it to a NumPy array.

Practice

1. What does the detach() method do to a PyTorch tensor?

easy

A. It creates a new tensor that shares data but is not tracked for gradients.

B. It deletes the tensor from memory immediately.

C. It converts the tensor to a NumPy array.

D. It resets the tensor's values to zero.

Solution

1. Step 1: Understand what detach() does

   The detach() method creates a new tensor that shares the same data but is removed from the computation graph, so gradients are not tracked.

2. Step 2: Compare options with detach() behavior

   Deleting the tensor from memory immediately, converting to a NumPy array, and resetting values to zero are all unrelated to detach(). Only creating a new tensor that shares data but is not tracked for gradients correctly describes the behavior.

3. Final Answer:

   It creates a new tensor that shares data but is not tracked for gradients. -> Option A

4. Quick Check:

   detach() = stops gradient tracking [OK]

Hint: detach() stops gradients but shares data memory [OK]

Common Mistakes:

• Thinking detach() deletes the tensor
• Confusing detach() with conversion to NumPy
• Assuming detach() resets tensor values

2. Which of the following is the correct syntax to detach a tensor x in PyTorch?

easy

A. x.detach()

B. detach(x)

C. x.detach

D. x.detach[]

Solution

1. Step 1: Recall the method call syntax in PyTorch

   Methods are called with parentheses, so detach() must be called as x.detach().

2. Step 2: Check each option

   x.detach() uses correct method call syntax. detach(x) is a function call which does not exist. x.detach misses parentheses, so it returns a method object, not a tensor. x.detach[] uses invalid syntax.

3. Final Answer:

   x.detach() -> Option A

4. Quick Check:

   Method call needs parentheses [OK]

Hint: Use parentheses to call detach() method [OK]

Common Mistakes:

• Forgetting parentheses after detach
• Using detach as a standalone function
• Using invalid bracket syntax

3. Consider the following code:

import torch
x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x * 2
y_detached = y.detach()
z = y_detached.sum()

What happens if you call z.backward()?

medium

A. It computes gradients for x successfully.

B. It raises an error because y_detached is detached from the graph.

C. It computes gradients for y_detached only.

D. It does nothing because z is a scalar.

Solution

1. Step 1: Understand detach() effect on computation graph

   Calling detach() on y creates a tensor not tracked for gradients, so y_detached has no gradient history.

2. Step 2: Analyze backward() on detached tensor

   Calling backward() on z (which depends on y_detached) will raise an error because y_detached is not connected to x or any requires_grad=True tensor.

3. Final Answer:

   It raises an error because y_detached is detached from the graph. -> Option B

4. Quick Check:

   Backward on detached tensor = error [OK]

Hint: Backward fails on detached tensors without grad history [OK]

Common Mistakes:

• Assuming backward computes gradients through detached tensors
• Thinking detached tensors still track gradients
• Ignoring that detach breaks gradient flow

4. You have the following code snippet:

import torch
x = torch.tensor([3.0], requires_grad=True)
y = x * 4
y_detached = y.detach()
z = y_detached + 2
y.backward()

What is the error in this code?

medium

A. Using detach() before backward() causes runtime error.

B. Calling backward() on y_detached instead of y.

C. Calling backward() on y which depends on detached tensor y_detached.

D. No error; code runs fine and computes gradients.

Solution

1. Step 1: Identify which tensor backward() is called on

   Backward is called on y, which is connected to x and requires gradients.

2. Step 2: Check if detach() affects backward on y

   Detach is called on y to create y_detached, but backward is not called on y_detached. So, gradient computation for y and x works fine.

3. Final Answer:

   No error; code runs fine and computes gradients. -> Option D

4. Quick Check:

   Backward on original tensor works despite detach() elsewhere [OK]

Hint: Backward on original tensor works even if detached tensor exists [OK]

Common Mistakes:

• Confusing which tensor backward() is called on
• Assuming detach() breaks all gradient computations
• Thinking detach() affects original tensor's graph

5. You want to save memory during training by stopping gradient tracking on intermediate tensor h but still use its values for further computation. Which approach correctly applies detach() in this scenario?

import torch
x = torch.tensor([2.0], requires_grad=True)
h = (x ** 2) + 3
# Your code here
y = h * 5
y.backward()

hard

A. Use h = h.clone() before computing y.

B. Call h.requires_grad_(False) instead of detach().

C. Replace h with h = h.detach() before computing y.

D. Do not use detach(); just call y.backward() directly.

Solution

1. Step 1: Understand the goal of stopping gradient tracking

   To save memory and stop gradients on h, we must detach it from the computation graph before using it further.

2. Step 2: Evaluate each option

   Replace h with h = h.detach() before computing y. correctly uses h.detach() to stop gradient tracking. Call h.requires_grad_(False) instead of detach(). changes requires_grad but does not detach from graph. Do not use detach(); just call y.backward() directly. keeps gradients tracked, not saving memory. Use h = h.clone() before computing y. clones data but does not stop gradient tracking.

3. Final Answer:

   Replace h with h.detach() before computing y. -> Option C

4. Quick Check:

   detach() stops gradients and saves memory [OK]

Hint: Use h = h.detach() to stop gradients but keep data [OK]

Common Mistakes:

• Using requires_grad_(False) instead of detach()
• Not detaching and expecting memory savings
• Using clone() which does not stop gradients