Label smoothing in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Label smoothing

Which metric matters for Label smoothing and WHY

Label smoothing helps the model avoid being too confident about its predictions. It softens the target labels, so the model learns better general patterns. The key metrics to watch are Cross-Entropy Loss and Accuracy. Cross-Entropy Loss shows how well the model predicts the smoothed labels, and Accuracy shows how often the model predicts the correct class. Because labels are softened, accuracy might be slightly lower but the model generalizes better.

Confusion matrix example with Label smoothing

    Actual \ Predicted | Class A | Class B | Class C
    ---------------------------------------------
    Class A           |   45    |   3     |   2
    Class B           |   4     |   43    |   3
    Class C           |   1     |   5     |   44

    Total samples = 150
    

From this matrix, we calculate metrics like precision and recall for each class. Label smoothing helps reduce overconfidence that can cause wrong predictions to be very confident.

Precision vs Recall tradeoff with Label smoothing

Label smoothing slightly lowers precision and recall because it softens the targets. This means the model is less sure about any single class, which can reduce false positives (improving precision) and false negatives (improving recall) in some cases.

For example, in a spam filter, label smoothing can help the model avoid marking too many good emails as spam (false positives), improving precision. But it might also miss some spam emails (false negatives), lowering recall a bit.

So, label smoothing balances precision and recall by preventing the model from being too confident, which helps in noisy or uncertain data.

What "good" vs "bad" metric values look like with Label smoothing

Good: Cross-Entropy Loss steadily decreases during training, and accuracy improves without sudden jumps. Precision and recall are balanced, showing the model is confident but not overconfident.

Bad: Very low loss but accuracy does not improve, or accuracy is high but the model fails on new data (overfitting). Precision or recall is very low, meaning the model is either too cautious or too confident on wrong classes.

Common pitfalls with Label smoothing metrics

• Accuracy paradox: Accuracy might be lower with label smoothing but the model is actually better at generalizing.
• Misinterpreting loss: Cross-Entropy Loss with label smoothing is different from normal loss, so comparing them directly can be misleading.
• Overfitting signs: If loss keeps decreasing but validation accuracy drops, the model might be memorizing smoothed labels instead of learning patterns.
• Ignoring class imbalance: Label smoothing does not fix class imbalance, so metrics like precision and recall per class are important.

Self-check question

Your model uses label smoothing and has 98% accuracy but only 12% recall on the fraud class. Is it good for production?

Answer: No, it is not good. Even with high accuracy, the very low recall means the model misses most fraud cases. For fraud detection, recall is critical because missing fraud is costly. Label smoothing helps generalize but does not fix low recall. You need to improve recall before production.

Key Result

Label smoothing improves model generalization by softening targets, balancing precision and recall, but requires careful interpretation of loss and accuracy.

Practice

1. What is the main purpose of label smoothing in PyTorch?

easy

A. To increase the learning rate automatically

B. To make the model less confident and improve generalization

C. To add noise to the input data

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand label smoothing concept

   Label smoothing softens the target labels, making the model less confident about the exact class.

2. Step 2: Connect to model behavior

   This helps the model generalize better by not being too sure, reducing overfitting.

3. Final Answer:

   To make the model less confident and improve generalization -> Option B

4. Quick Check:

   Label smoothing = less confident model [OK]

Hint: Label smoothing reduces confidence to improve generalization [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with data augmentation
• Assuming it reduces dataset size

2. Which of the following is the correct way to apply label smoothing in PyTorch's CrossEntropyLoss?

easy

A. loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

B. loss_fn = torch.nn.CrossEntropyLoss(smooth_labels=0.1)

C. loss_fn = torch.nn.CrossEntropyLoss(smoothing=0.1)

D. loss_fn = torch.nn.CrossEntropyLoss(label_smooth=0.1)

Solution

1. Step 1: Recall PyTorch CrossEntropyLoss parameters

   The correct parameter name for label smoothing is exactly 'label_smoothing'.

2. Step 2: Match correct syntax

   Only loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) uses the exact parameter name and value format.

3. Final Answer:

   loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) -> Option A

4. Quick Check:

   Parameter name is 'label_smoothing' [OK]

Hint: Use exact parameter name 'label_smoothing' in CrossEntropyLoss [OK]

Common Mistakes:

• Using incorrect parameter names like 'smooth_labels'
• Misspelling 'label_smoothing'
• Passing label smoothing outside loss function

3. Given the following code snippet, what will be the printed loss value trend when label smoothing is applied?

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.2)
logits = torch.tensor([[2.0, 0.5, 0.3]])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(round(loss.item(), 3))

medium

A. Loss will be negative

B. Loss will be zero

C. Loss will be lower than without label smoothing

D. Loss will be higher than without label smoothing

Solution

1. Step 1: Understand effect of label smoothing on loss

   Label smoothing softens the target, so the loss does not become zero even if prediction is perfect.

2. Step 2: Compare loss values

   Without smoothing, loss can be very low; with smoothing, loss is higher because targets are less certain.

3. Final Answer:

   Loss will be higher than without label smoothing -> Option D

4. Quick Check:

   Label smoothing increases loss value slightly [OK]

Hint: Label smoothing raises loss by softening targets [OK]

Common Mistakes:

• Expecting loss to be zero with smoothing
• Thinking smoothing lowers loss always
• Confusing loss sign (negative)

4. Identify the error in this PyTorch code snippet using label smoothing:

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
logits = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[2]])
loss = loss_fn(logits, target)
print(loss.item())

medium

A. Target tensor shape should be 1D, not 2D

B. Label smoothing parameter must be an integer

C. Logits tensor should be 1D, not 2D

D. CrossEntropyLoss does not support label smoothing

Solution

1. Step 1: Check target tensor shape

   CrossEntropyLoss expects target as 1D tensor of class indices, but target is 2D here.

2. Step 2: Confirm label smoothing usage

   Label smoothing parameter is correctly used as float; logits shape is correct as batch size 1 with 3 classes.

3. Final Answer:

   Target tensor shape should be 1D, not 2D -> Option A

4. Quick Check:

   Target shape must be 1D for CrossEntropyLoss [OK]

Hint: Target tensor must be 1D class indices [OK]

Common Mistakes:

• Passing target as 2D tensor
• Using integer for label_smoothing
• Misunderstanding CrossEntropyLoss support

5. You want to train a classification model with 5 classes using label smoothing of 0.1. Which of the following target label vectors correctly applies label smoothing manually for class 2 (index 1)?

hard

A. [0.2, 0.2, 0.2, 0.2, 0.2]

B. [0, 1, 0, 0, 0]

C. [0.025, 0.9, 0.025, 0.025, 0.025]

D. [0.1, 0.1, 0.1, 0.1, 0.6]

Solution

1. Step 1: Recall label smoothing formula

   With smoothing ε=0.1 and K=5 classes, true class gets 1 - ε = 0.9, each of the other K-1=4 classes gets ε / (K-1) = 0.1 / 4 = 0.025.

2. Step 2: Construct target for true class index 1

   The vector is [0.025, 0.9, 0.025, 0.025, 0.025].

3. Final Answer:

   [0.025, 0.9, 0.025, 0.025, 0.025] -> Option C

4. Quick Check:

   Smoothed target sums to 1 with 0.1 smoothing [OK]

Hint: Distribute smoothing evenly, reduce true class by smoothing [OK]

Common Mistakes:

• Using one-hot vector without smoothing
• Assigning smoothing incorrectly to true class
• Making all classes equal probability