Label smoothing in PyTorch

Introduction

Label smoothing helps the model not be too sure about its answers. It makes training more stable and can improve how well the model works on new data.

When training a classification model and you want to avoid overconfidence.

When your model tends to predict one class too strongly and ignores others.

When you want to improve the model's ability to generalize to new examples.

When your training labels might have some noise or errors.

When you want to reduce the chance of the model getting stuck on wrong answers.

Syntax

PyTorch

import torch.nn as nn

loss = nn.CrossEntropyLoss(label_smoothing=0.1)

The label_smoothing parameter takes a value between 0 and 1.

A value of 0 means no smoothing (normal labels), and higher values soften the labels more.

Examples

No label smoothing, normal one-hot labels.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.0)

Labels are smoothed by 10%, making the model less confident.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.1)

Stronger smoothing with 20%, useful if labels are noisy.

PyTorch

loss = nn.CrossEntropyLoss(label_smoothing=0.2)

Sample Model

This code trains a simple model on 3 samples with 3 classes using label smoothing of 0.1. It prints the loss each epoch and shows the predicted classes after training.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple dataset: 3 samples, 3 classes
inputs = torch.tensor([[1.0, 2.0, 3.0],
                       [1.0, 0.0, 0.0],
                       [0.0, 1.0, 0.0]])
labels = torch.tensor([2, 0, 1])  # correct classes

# Simple linear model
model = nn.Linear(3, 3)

# Loss with label smoothing
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Training loop for 5 epochs
for epoch in range(5):
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

# Predictions after training
with torch.no_grad():
    outputs = model(inputs)
    predicted = torch.argmax(outputs, dim=1)
    print("Predicted classes:", predicted.tolist())

Important Notes

Label smoothing changes the target labels from hard 0 or 1 to softer values like 0.9 and 0.05.

This helps the model avoid becoming too confident and can improve accuracy on new data.

Too much smoothing can make training harder, so choose a small value like 0.1 or 0.2.

Summary

Label smoothing makes labels less strict to help the model learn better.

It is easy to add in PyTorch using CrossEntropyLoss(label_smoothing=...).

Use it when you want your model to be less confident and more general.