Label smoothing in PyTorch - Deep Dive

Overview - Label smoothing

What is it?

Label smoothing is a technique used in training machine learning models to make the model less confident about its predictions. Instead of assigning a full probability of 1 to the correct class and 0 to others, it assigns a slightly lower probability to the correct class and distributes the remaining probability among the other classes. This helps the model generalize better and avoid overfitting.

Why it matters

Without label smoothing, models can become too confident about their predictions, which makes them less flexible and more likely to make big mistakes on new data. Label smoothing helps models stay humble and cautious, leading to better performance on real-world tasks where data can be noisy or different from training data.

Where it fits

Before learning label smoothing, you should understand basic classification tasks, how models output probabilities, and loss functions like cross-entropy. After mastering label smoothing, you can explore advanced regularization techniques and calibration methods to improve model reliability.

Mental Model

Core Idea

Label smoothing gently softens the target labels to prevent the model from becoming overly confident and to improve generalization.

Think of it like...

Imagine a teacher grading a test but instead of giving a perfect score for a correct answer, they give a slightly lower score to encourage students to stay curious and not assume they know everything perfectly.

┌───────────────────────────────┐
│ Original label:               │
│ Class A: 1.0                 │
│ Class B: 0.0                 │
│ Class C: 0.0                 │
└───────────────────────────────┘
          ↓ label smoothing
┌───────────────────────────────┐
│ Smoothed label:               │
│ Class A: 0.9                 │
│ Class B: 0.05                │
│ Class C: 0.05                │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding One-Hot Labels

Concept: One-hot encoding represents the correct class with a 1 and all others with 0 in classification tasks.

In classification, the true label is often represented as a vector where the correct class is 1 and all others are 0. For example, if there are three classes and the correct class is the first one, the label vector is [1, 0, 0]. This is called one-hot encoding.

Result

The model learns to predict a probability close to 1 for the correct class and 0 for others.

Understanding one-hot labels is essential because label smoothing modifies these labels to improve model training.

2

Foundation

Cross-Entropy Loss Basics

Concept: Cross-entropy loss measures how well the predicted probabilities match the true labels.

Cross-entropy loss compares the predicted probabilities from the model with the true labels. It penalizes the model more when it assigns low probability to the correct class and less when it assigns high probability. The goal is to minimize this loss during training.

Result

The model adjusts its predictions to reduce the loss, improving accuracy.

Knowing how cross-entropy works helps understand why changing labels affects training.

3

Intermediate

What Label Smoothing Does

🤔Before reading on: do you think label smoothing changes the model's predictions or the target labels? Commit to your answer.

Concept: Label smoothing changes the target labels by assigning less than 100% probability to the correct class and spreading the rest to other classes.

Instead of using [1, 0, 0] as the target, label smoothing might use [0.9, 0.05, 0.05]. This means the model is encouraged to be confident but not absolutely certain. This reduces overfitting and helps the model handle ambiguous or noisy data better.

Result

The model learns to avoid extreme confidence, leading to better generalization.

Understanding that label smoothing modifies targets, not predictions, clarifies its role as a regularizer.

4

Intermediate

Implementing Label Smoothing in PyTorch

🤔Before reading on: do you think label smoothing is built into PyTorch's loss functions or requires custom code? Commit to your answer.

Concept: PyTorch provides a built-in way to apply label smoothing in its cross-entropy loss function by setting a smoothing parameter.

In PyTorch, you can use torch.nn.CrossEntropyLoss with the parameter label_smoothing set to a value like 0.1. This automatically smooths the labels during loss calculation. For example: loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) This means the correct class label is reduced by 0.1 and the rest is spread evenly among other classes.

Result

The loss function applies label smoothing internally, simplifying training code.

Knowing built-in support saves time and reduces errors compared to manual label smoothing.

5

Intermediate

Effect on Model Confidence and Calibration

🤔Before reading on: does label smoothing increase or decrease model confidence? Commit to your answer.

Concept: Label smoothing reduces the model's confidence in its predictions, which can improve calibration and reduce overfitting.

Models trained with label smoothing tend to produce softer probability distributions, meaning they are less likely to assign near 100% probability to any class. This helps the model better reflect uncertainty and improves calibration, meaning predicted probabilities better match actual correctness likelihood.

Result

Improved model reliability and robustness on unseen data.

Understanding confidence reduction explains why label smoothing helps in real-world noisy environments.

6

Advanced

Label Smoothing and Gradient Behavior

🤔Before reading on: do you think label smoothing affects the gradients during backpropagation? Commit to your answer.

Concept: Label smoothing changes the target distribution, which alters the gradients and prevents the model from pushing probabilities to extremes.

During training, the loss gradient guides the model to adjust weights. With one-hot labels, the gradient pushes the model to predict 1 for the correct class and 0 for others. Label smoothing softens this push, resulting in smaller gradients near the extremes. This prevents the model from becoming overconfident and helps it learn more stable features.

Result

More stable training and better generalization.

Knowing how label smoothing affects gradients reveals its role as a subtle but powerful regularizer.

7

Expert

Surprising Effects and Limitations of Label Smoothing

🤔Before reading on: do you think label smoothing always improves accuracy? Commit to your answer.

Concept: Label smoothing can sometimes reduce the model's ability to learn fine distinctions and may hurt performance if used improperly.

While label smoothing helps generalization, it can also blur class boundaries, making it harder for the model to distinguish very similar classes. In tasks requiring very precise predictions, label smoothing might reduce accuracy. Also, it can interfere with techniques like knowledge distillation if not carefully tuned.

Result

Label smoothing is a tradeoff and must be applied thoughtfully.

Recognizing label smoothing's limits prevents misuse and guides better model design.

Under the Hood

Label smoothing works by modifying the target probability distribution used in the loss function. Instead of a hard 1 for the correct class and 0 for others, it assigns a value less than 1 to the correct class and distributes the remaining probability mass evenly among incorrect classes. This changes the cross-entropy loss landscape, resulting in softer gradients that discourage the model from becoming overly confident. Internally, during backpropagation, this leads to smaller gradient magnitudes near the extremes, promoting smoother weight updates and better generalization.

Why designed this way?

Label smoothing was designed to address overfitting and overconfidence in deep learning models. Traditional one-hot labels encourage models to assign full probability to a single class, which can cause sharp decision boundaries and poor calibration. By smoothing labels, the model learns to be less certain, which improves robustness to noise and unseen data. Alternatives like confidence penalty or entropy regularization exist, but label smoothing is simple, effective, and easy to integrate into existing loss functions.

┌───────────────────────────────┐
│ True label vector             │
│ [1, 0, 0, ..., 0]             │
└─────────────┬─────────────────┘
              │ label smoothing
              ▼
┌───────────────────────────────┐
│ Smoothed label vector          │
│ [1 - ε, ε/(K-1), ..., ε/(K-1)]│
└─────────────┬─────────────────┘
              │ used in
              ▼
┌───────────────────────────────┐
│ Cross-entropy loss function    │
│ computes loss and gradients    │
└─────────────┬─────────────────┘
              │ backpropagation
              ▼
┌───────────────────────────────┐
│ Model weight updates           │
│ smoother gradients prevent     │
│ overconfidence                │
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does label smoothing change the model's predicted probabilities directly? Commit to yes or no.

Common Belief: Label smoothing changes the model's output probabilities to be less confident.

Reality: Label smoothing changes the target labels used during training, not the model's predictions directly.

Why it matters: Confusing this leads to misunderstanding how label smoothing works and how to implement it correctly.

Quick: Does label smoothing always improve model accuracy? Commit to yes or no.

Common Belief: Label smoothing always makes the model more accurate.

Reality: Label smoothing can sometimes reduce accuracy, especially in tasks needing very precise class distinctions.

Why it matters: Assuming it always helps can cause performance drops if applied blindly.

Quick: Is label smoothing the same as adding noise to inputs? Commit to yes or no.

Common Belief: Label smoothing is a form of input noise or data augmentation.

Reality: Label smoothing modifies target labels, not inputs; it is a regularization on the output distribution.

Why it matters: Mixing these concepts can lead to incorrect training strategies.

Quick: Can label smoothing replace all other regularization methods? Commit to yes or no.

Common Belief: Label smoothing alone is enough to prevent overfitting.

Reality: Label smoothing helps but does not replace other regularization techniques like dropout or weight decay.

Why it matters: Overreliance on label smoothing can leave models vulnerable to overfitting.

Expert Zone

1

Label smoothing can interfere with knowledge distillation if the teacher model's soft targets are not adjusted accordingly.

2

The smoothing parameter ε should be tuned carefully; too high values can overly blur class boundaries and harm performance.

3

Label smoothing affects model calibration, often improving it, but may reduce the maximum achievable confidence for correct predictions.

When NOT to use

Avoid label smoothing in tasks requiring very sharp decision boundaries or when precise probability estimates are critical, such as medical diagnosis or risk assessment. Instead, consider alternatives like focal loss or confidence calibration methods.

Production Patterns

In production, label smoothing is commonly used in image classification and natural language processing to improve robustness. It is often combined with other regularizers like dropout and weight decay. Engineers tune the smoothing factor as a hyperparameter and monitor calibration metrics alongside accuracy.

Connections

Regularization

Label smoothing is a form of regularization that prevents overfitting by softening targets.

Understanding label smoothing as regularization helps connect it to techniques like dropout and weight decay that also improve generalization.

Calibration in Statistics

Label smoothing improves model calibration, making predicted probabilities better reflect true likelihoods.

Knowing calibration concepts from statistics clarifies why label smoothing leads to more reliable probability estimates.

Human Learning and Grading

Label smoothing mimics a grading style that avoids giving perfect scores to encourage cautious learning.

This connection shows how ideas from education psychology can inspire machine learning techniques.

Common Pitfalls

#1 Applying label smoothing by manually changing labels but forgetting to adjust the loss function accordingly.

Wrong approach:

labels = torch.tensor([[0.9, 0.05, 0.05]])
loss_fn = torch.nn.CrossEntropyLoss()
loss = loss_fn(predictions, labels) # This will error because CrossEntropyLoss expects class indices, not probabilities

Correct approach:

loss_fn = torch.nn.KLDivLoss(reduction='batchmean')
log_probs = torch.nn.functional.log_softmax(predictions, dim=1)
loss = loss_fn(log_probs, labels) # Use KL divergence with smoothed labels

Root cause:

Misunderstanding that CrossEntropyLoss expects integer class labels, not probability distributions.

#2 Setting label_smoothing parameter too high, e.g., 0.9, which makes the correct class almost indistinguishable.

Wrong approach:

loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.9)

Correct approach:

loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

Root cause:

Not tuning smoothing parameter properly, leading to excessive label softening and poor learning.

#3 Using label smoothing in regression tasks where labels are continuous values.

Wrong approach:

Applying label smoothing to continuous targets like [2.5, 3.0] in regression.

Correct approach:

Use label smoothing only in classification tasks with discrete classes.

Root cause:

Confusing classification label smoothing with regression targets.

Key Takeaways

Label smoothing modifies target labels to reduce model overconfidence and improve generalization.

It works by assigning less than full probability to the correct class and distributing the rest among others.

PyTorch supports label smoothing directly in its cross-entropy loss function for easy integration.

While helpful, label smoothing is not a cure-all and must be tuned carefully to avoid harming accuracy.

Understanding label smoothing's effect on gradients and calibration deepens insight into model training dynamics.