Label smoothing in PyTorch

Introduction

Label smoothing helps the model not be too sure about its answers. It makes training more stable and can improve how well the model works on new data.

When training a classification model and you want to avoid overconfidence.

When your model tends to predict one class too strongly and ignores others.

When you want to improve the model's ability to generalize to new examples.

When your training labels might have some noise or errors.

When you want to reduce the chance of the model getting stuck on wrong answers.

Syntax

PyTorch

import torch.nn as nn

loss = nn.CrossEntropyLoss(label_smoothing=0.1)

The label_smoothing parameter takes a value between 0 and 1.

A value of 0 means no smoothing (normal labels), and higher values soften the labels more.

Examples

No label smoothing, normal one-hot labels.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.0)

Labels are smoothed by 10%, making the model less confident.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.1)

Stronger smoothing with 20%, useful if labels are noisy.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.2)

Sample Model

This code trains a simple model on 3 samples with 3 classes using label smoothing of 0.1. It prints the loss each epoch and shows the predicted classes after training.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple dataset: 3 samples, 3 classes
inputs = torch.tensor([[1.0, 2.0, 3.0],
                       [1.0, 0.0, 0.0],
                       [0.0, 1.0, 0.0]])
labels = torch.tensor([2, 0, 1])  # correct classes

# Simple linear model
model = nn.Linear(3, 3)

# Loss with label smoothing
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Training loop for 5 epochs
for epoch in range(5):
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

# Predictions after training
with torch.no_grad():
    outputs = model(inputs)
    predicted = torch.argmax(outputs, dim=1)
    print("Predicted classes:", predicted.tolist())

Important Notes

Label smoothing changes the target labels from hard 0 or 1 to softer values like 0.9 and 0.05.

This helps the model avoid becoming too confident and can improve accuracy on new data.

Too much smoothing can make training harder, so choose a small value like 0.1 or 0.2.

Summary

Label smoothing makes labels less strict to help the model learn better.

It is easy to add in PyTorch using CrossEntropyLoss(label_smoothing=...).

Use it when you want your model to be less confident and more general.

Practice

1. What is the main purpose of label smoothing in PyTorch?

easy

A. To increase the learning rate automatically

B. To make the model less confident and improve generalization

C. To add noise to the input data

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand label smoothing concept

   Label smoothing softens the target labels, making the model less confident about the exact class.

2. Step 2: Connect to model behavior

   This helps the model generalize better by not being too sure, reducing overfitting.

3. Final Answer:

   To make the model less confident and improve generalization -> Option B

4. Quick Check:

   Label smoothing = less confident model [OK]

Hint: Label smoothing reduces confidence to improve generalization [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with data augmentation
• Assuming it reduces dataset size

2. Which of the following is the correct way to apply label smoothing in PyTorch's CrossEntropyLoss?

easy

A. loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

B. loss_fn = torch.nn.CrossEntropyLoss(smooth_labels=0.1)

C. loss_fn = torch.nn.CrossEntropyLoss(smoothing=0.1)

D. loss_fn = torch.nn.CrossEntropyLoss(label_smooth=0.1)

Solution

1. Step 1: Recall PyTorch CrossEntropyLoss parameters

   The correct parameter name for label smoothing is exactly 'label_smoothing'.

2. Step 2: Match correct syntax

   Only loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) uses the exact parameter name and value format.

3. Final Answer:

   loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) -> Option A

4. Quick Check:

   Parameter name is 'label_smoothing' [OK]

Hint: Use exact parameter name 'label_smoothing' in CrossEntropyLoss [OK]

Common Mistakes:

• Using incorrect parameter names like 'smooth_labels'
• Misspelling 'label_smoothing'
• Passing label smoothing outside loss function

3. Given the following code snippet, what will be the printed loss value trend when label smoothing is applied?

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.2)
logits = torch.tensor([[2.0, 0.5, 0.3]])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(round(loss.item(), 3))

medium

A. Loss will be negative

B. Loss will be zero

C. Loss will be lower than without label smoothing

D. Loss will be higher than without label smoothing

Solution

1. Step 1: Understand effect of label smoothing on loss

   Label smoothing softens the target, so the loss does not become zero even if prediction is perfect.

2. Step 2: Compare loss values

   Without smoothing, loss can be very low; with smoothing, loss is higher because targets are less certain.

3. Final Answer:

   Loss will be higher than without label smoothing -> Option D

4. Quick Check:

   Label smoothing increases loss value slightly [OK]

Hint: Label smoothing raises loss by softening targets [OK]

Common Mistakes:

• Expecting loss to be zero with smoothing
• Thinking smoothing lowers loss always
• Confusing loss sign (negative)

4. Identify the error in this PyTorch code snippet using label smoothing:

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
logits = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[2]])
loss = loss_fn(logits, target)
print(loss.item())

medium

A. Target tensor shape should be 1D, not 2D

B. Label smoothing parameter must be an integer

C. Logits tensor should be 1D, not 2D

D. CrossEntropyLoss does not support label smoothing

Solution

1. Step 1: Check target tensor shape

   CrossEntropyLoss expects target as 1D tensor of class indices, but target is 2D here.

2. Step 2: Confirm label smoothing usage

   Label smoothing parameter is correctly used as float; logits shape is correct as batch size 1 with 3 classes.

3. Final Answer:

   Target tensor shape should be 1D, not 2D -> Option A

4. Quick Check:

   Target shape must be 1D for CrossEntropyLoss [OK]

Hint: Target tensor must be 1D class indices [OK]

Common Mistakes:

• Passing target as 2D tensor
• Using integer for label_smoothing
• Misunderstanding CrossEntropyLoss support

5. You want to train a classification model with 5 classes using label smoothing of 0.1. Which of the following target label vectors correctly applies label smoothing manually for class 2 (index 1)?

hard

A. [0.2, 0.2, 0.2, 0.2, 0.2]

B. [0, 1, 0, 0, 0]

C. [0.025, 0.9, 0.025, 0.025, 0.025]

D. [0.1, 0.1, 0.1, 0.1, 0.6]

Solution

1. Step 1: Recall label smoothing formula

   With smoothing ε=0.1 and K=5 classes, true class gets 1 - ε = 0.9, each of the other K-1=4 classes gets ε / (K-1) = 0.1 / 4 = 0.025.

2. Step 2: Construct target for true class index 1

   The vector is [0.025, 0.9, 0.025, 0.025, 0.025].

3. Final Answer:

   [0.025, 0.9, 0.025, 0.025, 0.025] -> Option C

4. Quick Check:

   Smoothed target sums to 1 with 0.1 smoothing [OK]

Hint: Distribute smoothing evenly, reduce true class by smoothing [OK]

Common Mistakes:

• Using one-hot vector without smoothing
• Assigning smoothing incorrectly to true class
• Making all classes equal probability