Gradient clipping in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use gradient clipping in training?

Imagine you are training a neural network and notice the training loss suddenly spikes or the model weights become very large. Why would applying gradient clipping help in this situation?

A It prevents gradients from becoming too large, avoiding unstable updates and exploding gradients.

B It reduces the model size by pruning neurons with small gradients.

C It increases the learning rate automatically to speed up training.

D It normalizes the input data to have zero mean and unit variance.

💡 Hint

Think about what happens when gradients become very large during backpropagation.

Gradient clipping limits the size of gradients during training. This prevents very large updates that can cause the model to become unstable or diverge. It is especially useful in recurrent neural networks or deep networks where exploding gradients are common.

❓ Predict Output

intermediate

Output of gradient clipping code snippet

What will be the value of clipped_norm after running this PyTorch code?

PyTorch

import torch
from torch.nn.utils import clip_grad_norm_

model_params = [torch.nn.Parameter(torch.tensor([3.0, 4.0], requires_grad=True))]
for p in model_params:
    p.grad = torch.tensor([6.0, 8.0])

clipped_norm = clip_grad_norm_(model_params, max_norm=5.0)
print(round(clipped_norm.item(), 2))

A 14.0

B 5.0

C 1.0

D 10.0

💡 Hint

Calculate the norm of the original gradients before clipping.

The original gradient vector is [6, 8]. Its L2 norm is sqrt(6^2 + 8^2) = 10. The function returns the total norm before clipping, which is 10.0.

❓ Model Choice

advanced

Choosing when to apply gradient clipping

You are training two models: a shallow feedforward network and a deep recurrent neural network (RNN). Which model benefits more from gradient clipping and why?

A The deep RNN, because it has many layers and is prone to exploding gradients during backpropagation through time.

B The shallow feedforward network, because it has fewer layers and gradients can explode easily.

C Both models benefit equally from gradient clipping regardless of architecture.

D Neither model benefits from gradient clipping; it is only useful for convolutional networks.

💡 Hint

Consider which model type is more likely to have exploding gradients.

Deep RNNs often suffer from exploding gradients due to repeated multiplication of gradients through many time steps. Gradient clipping helps stabilize training in such models. Shallow feedforward networks usually do not have this problem.

❓ Hyperparameter

advanced

Effect of max_norm value in gradient clipping

In PyTorch's clip_grad_norm_, what happens if you set max_norm to a very small value like 0.1 during training?

A Gradients will be amplified to speed up training.

B Gradients will be ignored and training will proceed without updates.

C Gradients will be scaled down heavily, possibly slowing or stopping learning.

D The model will automatically increase the learning rate to compensate.

💡 Hint

Think about what happens when gradients are clipped to a very small norm.

If max_norm is too small, gradients get scaled down a lot, making weight updates very small. This can slow down or stall training because the model learns very slowly.

🔧 Debug

expert

Identifying error in gradient clipping usage

Consider this PyTorch training loop snippet. What error will it raise and why?

PyTorch

import torch
from torch.nn.utils import clip_grad_norm_

model = torch.nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.tensor([[1.0, 2.0]])
targets = torch.tensor([[1.0]])

optimizer.zero_grad()
outputs = model(inputs)
loss = torch.nn.functional.mse_loss(outputs, targets)
loss.backward()

clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

# Next iteration without zero_grad
outputs = model(inputs)
loss = torch.nn.functional.mse_loss(outputs, targets)
loss.backward()
clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

A ValueError: max_norm must be positive.

B No error; code runs fine.

C TypeError: clip_grad_norm_ expects a list of tensors.

D RuntimeError: Trying to backward through the graph a second time without retaining it.

💡 Hint

Look at the second backward call without clearing gradients.

No RuntimeError is raised because each loss.backward() call builds and backpropagates through a separate computational graph. However, omitting optimizer.zero_grad() before the second loss.backward() causes the new gradients to accumulate onto the stale gradients left over from the first iteration (after optimizer.step(), which does not clear gradients). This leads to incorrect parameter updates and is a common training loop bug.