Gradient clipping in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient clipping in machine learning?

Gradient clipping is a technique to limit or "clip" the gradients during training to prevent them from becoming too large, which helps avoid unstable updates and exploding gradients.

beginner

Why do exploding gradients cause problems during training?

Exploding gradients cause very large updates to model weights, which can make the training unstable and cause the model to fail to learn properly.

intermediate

How does PyTorch implement gradient clipping?

PyTorch provides functions like torch.nn.utils.clip_grad_norm_ and torch.nn.utils.clip_grad_value_ to clip gradients by norm or by value before the optimizer updates the model weights.

intermediate

What is the difference between clipping gradients by norm and by value?

Clipping by norm scales all gradients so their total length (norm) does not exceed a threshold, while clipping by value limits each individual gradient element to a maximum absolute value.

beginner

Show a simple PyTorch code snippet to clip gradients by norm.

After computing loss.backward(), use torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step() to clip gradients with max norm 1.0.

What problem does gradient clipping mainly solve?

A Exploding gradients

B Vanishing gradients

C Overfitting

D Underfitting

Gradient clipping is used to prevent exploding gradients by limiting their size.

Which PyTorch function clips gradients by their norm?

A torch.nn.utils.clip_grad_value_

B torch.clip_gradients

C torch.nn.utils.clip_grad_norm_

D torch.gradient_clip

torch.nn.utils.clip_grad_norm_ clips gradients based on their norm.

When should gradient clipping be applied during training?

A Before model initialization

B Before loss.backward()

C After optimizer.step()

D After loss.backward() and before optimizer.step()

Gradients are clipped after computing them with loss.backward() and before updating weights with optimizer.step().

Clipping gradients by value means:

A Limiting each gradient element to a max absolute value

B Scaling all gradients to have a fixed norm

C Setting all gradients to zero

D Increasing gradient values

Clipping by value limits each gradient element individually.

What happens if gradients are not clipped and explode?

A Model trains faster

B Training becomes unstable and may fail

C Model accuracy improves automatically

D Nothing changes

Exploding gradients cause unstable training and can prevent the model from learning.

Explain in your own words what gradient clipping is and why it is useful.

Think about what happens when gradients get too big during training.

Describe how to apply gradient clipping in a PyTorch training loop.

Remember the order of operations in training.

Practice

1. What is the main purpose of gradient clipping in PyTorch training?

easy

A. To prevent gradients from becoming too large and destabilizing training

B. To increase the learning rate automatically during training

C. To save memory by reducing model size

D. To initialize model weights before training

Solution

1. Step 1: Understand gradient behavior during training

   Gradients can sometimes become very large, causing unstable updates and training divergence.

2. Step 2: Role of gradient clipping

   Gradient clipping limits the size of gradients to keep training stable and prevent exploding gradients.

3. Final Answer:

   To prevent gradients from becoming too large and destabilizing training -> Option A

4. Quick Check:

   Gradient clipping = prevent large gradients [OK]

Hint: Gradient clipping stops gradients from exploding during training [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with weight initialization
• Believing it reduces model size

2. Which PyTorch function is used to clip gradients by their norm?

easy

A. torch.optim.clip_grad_norm

B. torch.nn.utils.clip_grad_value_

C. torch.nn.utils.clip_grad_norm_

D. torch.clip_gradients

Solution

1. Step 1: Recall PyTorch gradient clipping functions

   PyTorch provides two main functions: clip_grad_norm_ and clip_grad_value_ in torch.nn.utils.

2. Step 2: Identify function for norm clipping

   clip_grad_norm_ clips gradients based on their total norm, while clip_grad_value_ clips individual gradient values.

3. Final Answer:

   torch.nn.utils.clip_grad_norm_ -> Option C

4. Quick Check:

   Norm clipping function = clip_grad_norm_ [OK]

Hint: clip_grad_norm_ clips total gradient size by norm [OK]

Common Mistakes:

• Using clip_grad_value_ for norm clipping
• Assuming optimizer has clipping functions
• Using non-existent torch.clip_gradients

3. What will be the output of the following code snippet?

import torch
from torch.nn.utils import clip_grad_norm_

model = torch.nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

optimizer.zero_grad()
output = model(inputs)
loss = (output - target).pow(2).mean()
loss.backward()
clip_grad_norm_(model.parameters(), max_norm=0.1)
for p in model.parameters():
    print(p.grad.norm().item())

medium

A. Code will raise an error because clip_grad_norm_ is called before backward()

B. Gradient norms will be unchanged and possibly larger than 0.1

C. Gradients will be zero because of clipping

D. All printed gradient norms will be less than or equal to 0.1

Solution

1. Step 1: Understand code flow and gradient clipping

   Gradients are computed by loss.backward(), then clipped by clip_grad_norm_ with max_norm=0.1.

2. Step 2: Effect of clip_grad_norm_ on gradients

   clip_grad_norm_ rescales gradients so their total norm does not exceed 0.1, so printed norms will be ≤ 0.1.

3. Final Answer:

   All printed gradient norms will be less than or equal to 0.1 -> Option D

4. Quick Check:

   clip_grad_norm_ limits gradient norm ≤ max_norm [OK]

Hint: clip_grad_norm_ rescales gradients after backward [OK]

Common Mistakes:

• Calling clip_grad_norm_ before backward()
• Expecting gradients to be zero after clipping
• Thinking clipping increases gradient norms

4. Identify the error in this PyTorch training snippet using gradient clipping:

optimizer.zero_grad()
output = model(data)
loss = criterion(output, target)
clip_grad_norm_(model.parameters(), max_norm=1.0)
loss.backward()
optimizer.step()

medium

A. clip_grad_norm_ should be called after optimizer.step()

B. clip_grad_norm_ is called before loss.backward(), so gradients are not clipped

C. max_norm should be set to 0, not 1.0

D. optimizer.zero_grad() should be called after optimizer.step()

Solution

1. Step 1: Check order of operations for gradient clipping

   Gradients are created by loss.backward(), so clipping must happen after backward() to affect gradients.

2. Step 2: Identify mistake in code order

   clip_grad_norm_ is called before loss.backward(), so gradients do not exist yet and clipping has no effect.

3. Final Answer:

   clip_grad_norm_ is called before loss.backward(), so gradients are not clipped -> Option B

4. Quick Check:

   Clip gradients after backward() [OK]

Hint: Always clip gradients after backward(), before optimizer.step() [OK]

Common Mistakes:

• Clipping before backward()
• Calling zero_grad() after step()
• Setting max_norm to zero

5. You want to prevent exploding gradients in a deep RNN model. Which approach correctly applies gradient clipping in PyTorch during training?

hard

A. After loss.backward(), call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5), then optimizer.step()

B. Before loss.backward(), call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5), then optimizer.step()

C. After optimizer.step(), call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5)

D. Call torch.nn.utils.clip_grad_value_(model.parameters(), max_norm=5) before loss.backward()

Solution

1. Step 1: Understand correct gradient clipping sequence

   Gradients are computed by loss.backward(), so clipping must happen after this step and before optimizer.step().

2. Step 2: Identify correct function and order

   clip_grad_norm_ is used to clip by norm, suitable for RNNs to prevent exploding gradients. It must be called after backward() and before optimizer.step().

3. Final Answer:

   After loss.backward(), call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5), then optimizer.step() -> Option A

4. Quick Check:

   Clip gradients after backward(), before step() [OK]

Hint: Clip gradients after backward(), before optimizer step [OK]

Common Mistakes:

• Clipping before backward()
• Clipping after optimizer.step()
• Using clip_grad_value_ incorrectly