Gradient clipping in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient clipping in machine learning?

Gradient clipping is a technique to limit or "clip" the gradients during training to prevent them from becoming too large, which helps avoid unstable updates and exploding gradients.

beginner

Why do exploding gradients cause problems during training?

Exploding gradients cause very large updates to model weights, which can make the training unstable and cause the model to fail to learn properly.

intermediate

How does PyTorch implement gradient clipping?

PyTorch provides functions like torch.nn.utils.clip_grad_norm_ and torch.nn.utils.clip_grad_value_ to clip gradients by norm or by value before the optimizer updates the model weights.

intermediate

What is the difference between clipping gradients by norm and by value?

Clipping by norm scales all gradients so their total length (norm) does not exceed a threshold, while clipping by value limits each individual gradient element to a maximum absolute value.

beginner

Show a simple PyTorch code snippet to clip gradients by norm.

After computing loss.backward(), use torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step() to clip gradients with max norm 1.0.

What problem does gradient clipping mainly solve?

A Exploding gradients

B Vanishing gradients

C Overfitting

D Underfitting

Gradient clipping is used to prevent exploding gradients by limiting their size.

Which PyTorch function clips gradients by their norm?

A torch.nn.utils.clip_grad_value_

B torch.clip_gradients

C torch.nn.utils.clip_grad_norm_

D torch.gradient_clip

torch.nn.utils.clip_grad_norm_ clips gradients based on their norm.

When should gradient clipping be applied during training?

A Before model initialization

B Before loss.backward()

C After optimizer.step()

D After loss.backward() and before optimizer.step()

Gradients are clipped after computing them with loss.backward() and before updating weights with optimizer.step().

Clipping gradients by value means:

A Limiting each gradient element to a max absolute value

B Scaling all gradients to have a fixed norm

C Setting all gradients to zero

D Increasing gradient values

Clipping by value limits each gradient element individually.

What happens if gradients are not clipped and explode?

A Model trains faster

B Training becomes unstable and may fail

C Model accuracy improves automatically

D Nothing changes

Exploding gradients cause unstable training and can prevent the model from learning.

Explain in your own words what gradient clipping is and why it is useful.

Think about what happens when gradients get too big during training.

Describe how to apply gradient clipping in a PyTorch training loop.

Remember the order of operations in training.