Gradient clipping in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to clip gradients by norm before optimizer step.

PyTorch

torch.nn.utils.clip_grad_norm_(model.parameters(), [1])
optimizer.step()

Drag options to blanks, or click blank then click option'

A0.1

B10.0

C1.0

D0.01

Attempts:

3 left

💡 Hint

Common Mistakes

Using zero or negative values for max norm.

Using very large values that don't clip gradients effectively.

Gradient clipping by norm is commonly done with a max norm value like 1.0 to prevent exploding gradients.

2fill in blank

medium

Complete the code to clip gradients by value before optimizer step.

PyTorch

torch.nn.utils.clip_grad_value_(model.parameters(), [1])
optimizer.step()

Drag options to blanks, or click blank then click option'

A5

B0.5

C0.05

D10

Attempts:

3 left

💡 Hint

Common Mistakes

Using values too small that zero out gradients.

Using negative values which cause errors.

Gradient clipping by value limits each gradient element to a max absolute value, often a small integer like 5.

3fill in blank

hard

Fix the error in the code to properly clip gradients by norm.

PyTorch

torch.nn.utils.clip_grad_norm_(model.parameters(), [1])
optimizer.step()

Drag options to blanks, or click blank then click option'

A1.0

Bmax_norm

Cmax_norm=1.0

Dclip_value

Attempts:

3 left

💡 Hint

Common Mistakes

Passing variable names instead of values.

Using keyword arguments incorrectly.

The function expects a float value for max norm, not a variable name or keyword argument.

4fill in blank

hard

Fill both blanks to clip gradients by norm and then zero gradients.

PyTorch

torch.nn.utils.clip_grad_norm_(model.parameters(), [1])
optimizer.[2]()

Drag options to blanks, or click blank then click option'

A1.0

Bzero_grad

Cstep

Dclip_grad_norm_

Attempts:

3 left

💡 Hint

Common Mistakes

Calling step() before zero_grad().

Using wrong method names.

First clip gradients by norm 1.0, then clear gradients with zero_grad() before next step.

5fill in blank

hard

Fill all three blanks to clip gradients by value, perform optimizer step, and zero gradients.

PyTorch

torch.nn.utils.clip_grad_value_(model.parameters(), [1])
optimizer.[2]()
optimizer.[3]()

Drag options to blanks, or click blank then click option'

A0.5

Bstep

Czero_grad

Dclip_grad_value_

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up step and zero_grad order.

Using wrong clip values.

Clip gradients by value 0.5, then call optimizer.step() to update weights, and zero_grad() to clear gradients.