Detaching from computation graph in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this PyTorch code snippet?

Consider the following PyTorch code that creates a tensor, performs an operation, and detaches it from the computation graph. What will be the value of detached_tensor.requires_grad?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
detached_tensor = y.detach()
print(detached_tensor.requires_grad)

A False

B Raises an error

C True

D None

💡 Hint

Think about what detaching a tensor means for gradient tracking.

Detaching a tensor creates a new tensor that shares the same data but is not tracked by the computation graph. Therefore, requires_grad becomes False.

🧠 Conceptual

intermediate

Why do we use detach() in PyTorch?

Which of the following best explains the purpose of using detach() on a tensor in PyTorch?

A To create a tensor that shares data but does not track operations for gradient computation

B To permanently delete the tensor from memory

C To convert a tensor to a NumPy array

D To enable gradient tracking on a tensor that previously did not require gradients

💡 Hint

Think about how detaching affects the computation graph and gradient flow.

detach() creates a new tensor that shares the same data but is excluded from the computation graph, so no gradients are computed for it.

🔧 Debug

advanced

Why does this code raise an error when calling backward()?

Examine the code below. Why does calling loss.backward() raise an error?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2
detached_y = y.detach()
loss = detached_y.sum()
loss.backward()

A Because <code>loss</code> is not a scalar tensor

B Because <code>detached_y</code> is not connected to the computation graph, so gradients cannot be computed

C Because <code>x</code> does not require gradients

D Because <code>detach()</code> modifies the original tensor in-place

💡 Hint

Consider what happens to the graph when you detach a tensor and then try to backpropagate.

Detaching y removes it from the computation graph. Therefore, loss is not connected to x, and calling backward() raises an error because there is no graph to backpropagate through.

❓ Model Choice

advanced

When should you use detach() in a training loop?

Which scenario below correctly describes when to use detach() in a PyTorch training loop?

A When you want to initialize model weights randomly

B When you want to convert tensors to CPU for faster computation

C When you want to increase the learning rate dynamically

D When you want to stop gradients from flowing through part of the model to save memory or prevent updates

💡 Hint

Think about controlling which parts of the model get updated during training.

detach() is used to stop gradient flow through certain tensors, which can be useful to freeze parts of a model or save memory during backpropagation.

❓ Metrics

expert

How does detaching affect gradient computation and memory usage?

Consider a deep neural network where you detach intermediate tensors during forward pass. What is the effect on gradient computation and memory usage?

A Detaching intermediate tensors increases memory usage and speeds up gradient computation

B Detaching intermediate tensors has no effect on memory or gradients

C Detaching intermediate tensors reduces memory usage but prevents gradients from flowing back through those tensors, possibly affecting training

D Detaching intermediate tensors causes the model to train faster without any side effects

💡 Hint

Think about the trade-off between saving memory and allowing gradients to flow.

Detaching tensors cuts off gradient flow, so gradients are not computed for those parts, which saves memory but can affect training effectiveness.