Label smoothing in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is label smoothing in machine learning?

Label smoothing is a technique that softens the target labels by assigning a small probability to all classes instead of a hard 0 or 1. This helps the model avoid becoming too confident and improves generalization.

beginner

Why do we use label smoothing during training?

We use label smoothing to prevent the model from becoming overconfident on training data. It reduces overfitting and helps the model perform better on new, unseen data.

intermediate

How does label smoothing change the target labels?

Instead of using 1 for the correct class and 0 for others, label smoothing assigns a value like 0.9 to the correct class and distributes 0.1 among the other classes evenly.

intermediate

Show a simple PyTorch code snippet to apply label smoothing with CrossEntropyLoss.

You can use PyTorch's built-in label smoothing by setting the 'label_smoothing' parameter in CrossEntropyLoss, like this:<br><pre>import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)</pre>

advanced

What effect does label smoothing have on model confidence and calibration?

Label smoothing reduces the model's confidence in its predictions, which often leads to better calibrated probabilities and less overconfident wrong predictions.

What does label smoothing do to the target labels?

A Assigns a small positive value to all classes instead of hard 0 or 1

B Increases the learning rate during training

C Removes noisy data from the dataset

D Changes the model architecture

Label smoothing assigns a small positive value to all classes, softening the target labels.

Which PyTorch loss function parameter enables label smoothing?

A smooth_factor

B label_smoothing

C smooth_labels

D smoothing_rate

The 'label_smoothing' parameter in CrossEntropyLoss enables label smoothing.

What is a common benefit of using label smoothing?

A Larger model size

B Faster training speed

C Better model calibration and less overfitting

D More complex model architecture

Label smoothing helps the model generalize better by reducing overfitting and improving calibration.

If label smoothing is set to 0.1, what label value might the correct class get?

A 0.0

B 1.0

C 0.1

D 0.9

With 0.1 smoothing, the correct class label is reduced from 1.0 to 0.9.

Label smoothing is mainly used to:

A Prevent the model from becoming too confident

B Make the model more confident

C Increase the number of classes

D Make labels harder for the model

Label smoothing prevents the model from becoming too confident by softening the labels.

Explain what label smoothing is and why it helps improve model training.

Think about how changing the target labels affects model confidence.

Describe how to implement label smoothing in PyTorch using CrossEntropyLoss.

Check PyTorch documentation for CrossEntropyLoss parameters.

Practice

1. What is the main purpose of label smoothing in PyTorch?

easy

A. To increase the learning rate automatically

B. To make the model less confident and improve generalization

C. To add noise to the input data

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand label smoothing concept

   Label smoothing softens the target labels, making the model less confident about the exact class.

2. Step 2: Connect to model behavior

   This helps the model generalize better by not being too sure, reducing overfitting.

3. Final Answer:

   To make the model less confident and improve generalization -> Option B

4. Quick Check:

   Label smoothing = less confident model [OK]

Hint: Label smoothing reduces confidence to improve generalization [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with data augmentation
• Assuming it reduces dataset size

2. Which of the following is the correct way to apply label smoothing in PyTorch's CrossEntropyLoss?

easy

A. loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

B. loss_fn = torch.nn.CrossEntropyLoss(smooth_labels=0.1)

C. loss_fn = torch.nn.CrossEntropyLoss(smoothing=0.1)

D. loss_fn = torch.nn.CrossEntropyLoss(label_smooth=0.1)

Solution

1. Step 1: Recall PyTorch CrossEntropyLoss parameters

   The correct parameter name for label smoothing is exactly 'label_smoothing'.

2. Step 2: Match correct syntax

   Only loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) uses the exact parameter name and value format.

3. Final Answer:

   loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) -> Option A

4. Quick Check:

   Parameter name is 'label_smoothing' [OK]

Hint: Use exact parameter name 'label_smoothing' in CrossEntropyLoss [OK]

Common Mistakes:

• Using incorrect parameter names like 'smooth_labels'
• Misspelling 'label_smoothing'
• Passing label smoothing outside loss function

3. Given the following code snippet, what will be the printed loss value trend when label smoothing is applied?

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.2)
logits = torch.tensor([[2.0, 0.5, 0.3]])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(round(loss.item(), 3))

medium

A. Loss will be negative

B. Loss will be zero

C. Loss will be lower than without label smoothing

D. Loss will be higher than without label smoothing

Solution

1. Step 1: Understand effect of label smoothing on loss

   Label smoothing softens the target, so the loss does not become zero even if prediction is perfect.

2. Step 2: Compare loss values

   Without smoothing, loss can be very low; with smoothing, loss is higher because targets are less certain.

3. Final Answer:

   Loss will be higher than without label smoothing -> Option D

4. Quick Check:

   Label smoothing increases loss value slightly [OK]

Hint: Label smoothing raises loss by softening targets [OK]

Common Mistakes:

• Expecting loss to be zero with smoothing
• Thinking smoothing lowers loss always
• Confusing loss sign (negative)

4. Identify the error in this PyTorch code snippet using label smoothing:

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
logits = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[2]])
loss = loss_fn(logits, target)
print(loss.item())

medium

A. Target tensor shape should be 1D, not 2D

B. Label smoothing parameter must be an integer

C. Logits tensor should be 1D, not 2D

D. CrossEntropyLoss does not support label smoothing

Solution

1. Step 1: Check target tensor shape

   CrossEntropyLoss expects target as 1D tensor of class indices, but target is 2D here.

2. Step 2: Confirm label smoothing usage

   Label smoothing parameter is correctly used as float; logits shape is correct as batch size 1 with 3 classes.

3. Final Answer:

   Target tensor shape should be 1D, not 2D -> Option A

4. Quick Check:

   Target shape must be 1D for CrossEntropyLoss [OK]

Hint: Target tensor must be 1D class indices [OK]

Common Mistakes:

• Passing target as 2D tensor
• Using integer for label_smoothing
• Misunderstanding CrossEntropyLoss support

5. You want to train a classification model with 5 classes using label smoothing of 0.1. Which of the following target label vectors correctly applies label smoothing manually for class 2 (index 1)?

hard

A. [0.2, 0.2, 0.2, 0.2, 0.2]

B. [0, 1, 0, 0, 0]

C. [0.025, 0.9, 0.025, 0.025, 0.025]

D. [0.1, 0.1, 0.1, 0.1, 0.6]

Solution

1. Step 1: Recall label smoothing formula

   With smoothing ε=0.1 and K=5 classes, true class gets 1 - ε = 0.9, each of the other K-1=4 classes gets ε / (K-1) = 0.1 / 4 = 0.025.

2. Step 2: Construct target for true class index 1

   The vector is [0.025, 0.9, 0.025, 0.025, 0.025].

3. Final Answer:

   [0.025, 0.9, 0.025, 0.025, 0.025] -> Option C

4. Quick Check:

   Smoothed target sums to 1 with 0.1 smoothing [OK]

Hint: Distribute smoothing evenly, reduce true class by smoothing [OK]

Common Mistakes:

• Using one-hot vector without smoothing
• Assigning smoothing incorrectly to true class
• Making all classes equal probability