Weight decay (L2 regularization) in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to add L2 regularization (weight decay) to the optimizer.

PyTorch

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=[1])

Drag options to blanks, or click blank then click option'

A0.001

B1

C0

D-0.01

Attempts:

3 left

💡 Hint

Common Mistakes

Setting weight_decay to zero disables L2 regularization.

Using a negative value causes an error.

Weight decay is set by the weight_decay parameter in the optimizer. A small positive value like 0.001 applies L2 regularization.

2fill in blank

medium

Complete the code to create an SGD optimizer with learning rate 0.1 and weight decay 0.01.

PyTorch

optimizer = torch.optim.SGD(model.parameters(), lr=[1], weight_decay=[2])

Drag options to blanks, or click blank then click option'

A0.01

B1

C0.001

D0.1

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing learning rate with weight decay.

Using too large or too small learning rate.

The learning rate is set with lr to 0.1 (D) and weight decay with weight_decay to 0.01 (A).

3fill in blank

hard

Fix the error in the optimizer creation by filling the correct weight decay value.

PyTorch

optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=[1])

Drag options to blanks, or click blank then click option'

A-0.01

B0.0

C0.01

D"0.01"

Attempts:

3 left

💡 Hint

Common Mistakes

Using negative values for weight decay.

Passing weight decay as a string instead of a float.

Weight decay must be a positive float number. Negative or string values cause errors.

4fill in blank

hard

Fill both blanks to create an Adam optimizer with learning rate 0.0005 and weight decay 0.0001.

PyTorch

optimizer = torch.optim.Adam(model.parameters(), lr=[1], weight_decay=[2])

Drag options to blanks, or click blank then click option'

A0.0005

B0.001

C0.0001

D0.01

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping learning rate and weight decay values.

Using too large weight decay.

The learning rate is 0.0005 and weight decay is 0.0001 as specified.

5fill in blank

hard

Fill all three blanks to create an SGD optimizer with momentum 0.9, learning rate 0.01, and weight decay 0.0005.

PyTorch

optimizer = torch.optim.SGD(model.parameters(), momentum=[1], lr=[2], weight_decay=[3])

Drag options to blanks, or click blank then click option'

A0.9

B0.01

C0.0005

D0.1

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up momentum and learning rate values.

Using weight decay values that are too large.

Momentum is 0.9, learning rate is 0.01, and weight decay is 0.0005 as required.