Label smoothing in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is label smoothing in machine learning?

Label smoothing is a technique that softens the target labels by assigning a small probability to all classes instead of a hard 0 or 1. This helps the model avoid becoming too confident and improves generalization.

beginner

Why do we use label smoothing during training?

We use label smoothing to prevent the model from becoming overconfident on training data. It reduces overfitting and helps the model perform better on new, unseen data.

intermediate

How does label smoothing change the target labels?

Instead of using 1 for the correct class and 0 for others, label smoothing assigns a value like 0.9 to the correct class and distributes 0.1 among the other classes evenly.

intermediate

Show a simple PyTorch code snippet to apply label smoothing with CrossEntropyLoss.

You can use PyTorch's built-in label smoothing by setting the 'label_smoothing' parameter in CrossEntropyLoss, like this:<br><pre>import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)</pre>

advanced

What effect does label smoothing have on model confidence and calibration?

Label smoothing reduces the model's confidence in its predictions, which often leads to better calibrated probabilities and less overconfident wrong predictions.

What does label smoothing do to the target labels?

A Assigns a small positive value to all classes instead of hard 0 or 1

B Increases the learning rate during training

C Removes noisy data from the dataset

D Changes the model architecture

Label smoothing assigns a small positive value to all classes, softening the target labels.

Which PyTorch loss function parameter enables label smoothing?

A smooth_factor

B label_smoothing

C smooth_labels

D smoothing_rate

The 'label_smoothing' parameter in CrossEntropyLoss enables label smoothing.

What is a common benefit of using label smoothing?

A Larger model size

B Faster training speed

C Better model calibration and less overfitting

D More complex model architecture

Label smoothing helps the model generalize better by reducing overfitting and improving calibration.

If label smoothing is set to 0.1, what label value might the correct class get?

A 0.0

B 1.0

C 0.1

D 0.9

With 0.1 smoothing, the correct class label is reduced from 1.0 to 0.9.

Label smoothing is mainly used to:

A Prevent the model from becoming too confident

B Make the model more confident

C Increase the number of classes

D Make labels harder for the model

Label smoothing prevents the model from becoming too confident by softening the labels.

Explain what label smoothing is and why it helps improve model training.

Think about how changing the target labels affects model confidence.

Describe how to implement label smoothing in PyTorch using CrossEntropyLoss.

Check PyTorch documentation for CrossEntropyLoss parameters.