Why automatic differentiation enables training in PyTorch - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is automatic differentiation important in training neural networks?

Imagine you want to teach a robot to recognize cats in photos. The robot uses a neural network that learns by adjusting numbers called weights. Why do we need automatic differentiation for this learning process?

A It calculates the exact changes needed for weights by computing gradients automatically.

B It randomly changes weights without any calculation to speed up training.

C It stores all training data to use later for predictions.

D It converts images into text so the robot can understand them.

💡 Hint

Think about how the robot knows how to improve itself after seeing mistakes.

Automatic differentiation helps calculate gradients, which tell the model how to adjust weights to reduce errors. This is essential for training neural networks efficiently.

❓ Predict Output

intermediate

Output of PyTorch gradient calculation

What is the output of the following PyTorch code snippet?

PyTorch

import torch
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
z = y + 2 * x
z.backward()
print(x.grad.item())

A 12.0

B 14.0

C 10.0

D 8.0

💡 Hint

Recall the derivative of z = x^3 + 2x with respect to x.

The derivative dz/dx = 3x^2 + 2. At x=2, dz/dx = 3*(2^2) + 2 = 12 + 2 = 14.

❓ Model Choice

advanced

Choosing the right model for automatic differentiation demonstration

You want to demonstrate automatic differentiation with a simple model that has a clear gradient. Which model is best?

A A linear model with one input and one output: y = wx + b

B A random forest with multiple decision trees

C A k-nearest neighbors model without parameters

D A clustering model like k-means

💡 Hint

Think about which model has parameters that can be differentiated easily.

Linear models have parameters (weights and bias) that are differentiable, making them ideal to show automatic differentiation. Other models do not use gradient-based training.

❓ Hyperparameter

advanced

Effect of learning rate on gradient-based training

During training with automatic differentiation, you adjust the learning rate. What happens if the learning rate is too high?

A The gradients will become zero and stop updating weights.

B The model will train slower but more accurately.

C The model weights may change too much, causing training to become unstable.

D The model will ignore the gradients and use random updates.

💡 Hint

Consider what happens if you take very large steps when trying to reach a target.

A high learning rate causes large weight updates, which can overshoot the minimum and make training unstable or diverge.

🔧 Debug

expert

Why does this PyTorch code raise an error during backward pass?

Consider this PyTorch code:

import torch
x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
z = y.detach() + 1
z.backward()

What error will occur and why?

A ValueError because backward() requires a scalar but y is not scalar.

B TypeError because detach() returns a list instead of a tensor.

C No error; the code runs and prints gradients correctly.

D RuntimeError because y.detach() breaks the computation graph, so z has no grad_fn.

💡 Hint

Think about what detach() does to the tensor in terms of gradient tracking.

detach() creates a tensor that does not track gradients, breaking the graph. Calling backward() on z after using detach() on part of the graph causes a RuntimeError.