Label smoothing in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of label smoothing in classification?

Label smoothing is a technique used during training classification models. What is its main purpose?

A To reduce overconfidence of the model by softening the target labels

B To increase the learning rate dynamically during training

C To add noise to the input data for better generalization

D To convert multi-class problems into multiple binary problems

💡 Hint

Think about how label smoothing changes the target labels.

Label smoothing replaces hard 0 and 1 labels with softer values to prevent the model from becoming too confident, which helps generalization.

❓ Predict Output

intermediate

What is the output tensor after applying label smoothing?

Given the following PyTorch code applying label smoothing to a batch of 3 samples with 4 classes, what is the smoothed target tensor?

PyTorch

import torch

def smooth_labels(labels, smoothing=0.1, num_classes=4):
    with torch.no_grad():
        smooth_value = smoothing / num_classes
        one_hot = torch.zeros(labels.size(0), num_classes)
        one_hot.scatter_(1, labels.unsqueeze(1), 1.0)
        return one_hot * (1 - smoothing) + smooth_value

labels = torch.tensor([0, 2, 3])
smoothed = smooth_labels(labels)
print(smoothed)

A [[1, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

B [[0.9, 0.1, 0.1, 0.1], [0.1, 0.1, 0.9, 0.1], [0.1, 0.1, 0.1, 0.9]]

C [[0.925, 0.025, 0.025, 0.025], [0.025, 0.025, 0.925, 0.025], [0.025, 0.025, 0.025, 0.925]]

D [[0.9, 0.0333, 0.0333, 0.0333], [0.0333, 0.0333, 0.9, 0.0333], [0.0333, 0.0333, 0.0333, 0.9]]

💡 Hint

Calculate smoothing value as smoothing/num_classes and add it to all classes after scaling the one-hot by (1-smoothing).

With smoothing=0.1 and 4 classes, smoothing value is 0.1/4=0.025. The one-hot is scaled by (1-0.1)=0.9, then 0.025 is added to all, resulting in 0.925 for target and 0.025 for others.

❓ Hyperparameter

advanced

Choosing the label smoothing factor

Which label smoothing factor is most likely to cause the model to underfit?

A 0.5

B 0.05

C 0.0 (no smoothing)

D 0.1

💡 Hint

Think about what happens if the smoothing factor is very large.

A very high smoothing factor like 0.5 makes the target labels too soft, reducing the model's ability to learn clear distinctions, leading to underfitting.

❓ Metrics

advanced

Effect of label smoothing on training loss

When using label smoothing during training, how does the training loss typically behave compared to training without label smoothing?

A Training loss fluctuates randomly and unpredictably

B Training loss is lower because the model fits the smoothed labels better

C Training loss is the same because label smoothing does not affect loss values

D Training loss is higher because the model is penalized for being too confident

💡 Hint

Consider how smoothing changes the target labels and model confidence.

Label smoothing prevents the model from assigning full probability to the correct class, so the loss is higher since the model is penalized for overconfidence.

🔧 Debug

expert

Identify the error in this label smoothing implementation

What error will this PyTorch code raise when applying label smoothing?

PyTorch

import torch

def label_smooth(targets, smoothing=0.1, num_classes=5):
    smooth_val = smoothing / num_classes
    one_hot = torch.zeros(targets.size(0), num_classes)
    one_hot.scatter_(1, targets.unsqueeze(1), 1.0)
    return one_hot * (1 - smoothing) + smooth_val

targets = torch.tensor([1, 3, 4])
smoothed = label_smooth(targets)
print(smoothed)

A RuntimeError due to scatter_ index out of range

B No error, outputs smoothed labels correctly

C TypeError because smoothing is divided by num_classes instead of num_classes - 1

D ValueError because targets tensor shape is incompatible

💡 Hint

Check the smoothing value calculation and scatter_ usage carefully.

The code correctly implements a common variant of label smoothing using smoothing/num_classes, and scatter_ is used properly with valid indices (0 to 4 for num_classes=5). No error occurs.

Practice

1. What is the main purpose of label smoothing in PyTorch?

easy

A. To increase the learning rate automatically

B. To make the model less confident and improve generalization

C. To add noise to the input data

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand label smoothing concept

   Label smoothing softens the target labels, making the model less confident about the exact class.

2. Step 2: Connect to model behavior

   This helps the model generalize better by not being too sure, reducing overfitting.

3. Final Answer:

   To make the model less confident and improve generalization -> Option B

4. Quick Check:

   Label smoothing = less confident model [OK]

Hint: Label smoothing reduces confidence to improve generalization [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with data augmentation
• Assuming it reduces dataset size

2. Which of the following is the correct way to apply label smoothing in PyTorch's CrossEntropyLoss?

easy

A. loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

B. loss_fn = torch.nn.CrossEntropyLoss(smooth_labels=0.1)

C. loss_fn = torch.nn.CrossEntropyLoss(smoothing=0.1)

D. loss_fn = torch.nn.CrossEntropyLoss(label_smooth=0.1)

Solution

1. Step 1: Recall PyTorch CrossEntropyLoss parameters

   The correct parameter name for label smoothing is exactly 'label_smoothing'.

2. Step 2: Match correct syntax

   Only loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) uses the exact parameter name and value format.

3. Final Answer:

   loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) -> Option A

4. Quick Check:

   Parameter name is 'label_smoothing' [OK]

Hint: Use exact parameter name 'label_smoothing' in CrossEntropyLoss [OK]

Common Mistakes:

• Using incorrect parameter names like 'smooth_labels'
• Misspelling 'label_smoothing'
• Passing label smoothing outside loss function

3. Given the following code snippet, what will be the printed loss value trend when label smoothing is applied?

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.2)
logits = torch.tensor([[2.0, 0.5, 0.3]])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(round(loss.item(), 3))

medium

A. Loss will be negative

B. Loss will be zero

C. Loss will be lower than without label smoothing

D. Loss will be higher than without label smoothing

Solution

1. Step 1: Understand effect of label smoothing on loss

   Label smoothing softens the target, so the loss does not become zero even if prediction is perfect.

2. Step 2: Compare loss values

   Without smoothing, loss can be very low; with smoothing, loss is higher because targets are less certain.

3. Final Answer:

   Loss will be higher than without label smoothing -> Option D

4. Quick Check:

   Label smoothing increases loss value slightly [OK]

Hint: Label smoothing raises loss by softening targets [OK]

Common Mistakes:

• Expecting loss to be zero with smoothing
• Thinking smoothing lowers loss always
• Confusing loss sign (negative)

4. Identify the error in this PyTorch code snippet using label smoothing:

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
logits = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[2]])
loss = loss_fn(logits, target)
print(loss.item())

medium

A. Target tensor shape should be 1D, not 2D

B. Label smoothing parameter must be an integer

C. Logits tensor should be 1D, not 2D

D. CrossEntropyLoss does not support label smoothing

Solution

1. Step 1: Check target tensor shape

   CrossEntropyLoss expects target as 1D tensor of class indices, but target is 2D here.

2. Step 2: Confirm label smoothing usage

   Label smoothing parameter is correctly used as float; logits shape is correct as batch size 1 with 3 classes.

3. Final Answer:

   Target tensor shape should be 1D, not 2D -> Option A

4. Quick Check:

   Target shape must be 1D for CrossEntropyLoss [OK]

Hint: Target tensor must be 1D class indices [OK]

Common Mistakes:

• Passing target as 2D tensor
• Using integer for label_smoothing
• Misunderstanding CrossEntropyLoss support

5. You want to train a classification model with 5 classes using label smoothing of 0.1. Which of the following target label vectors correctly applies label smoothing manually for class 2 (index 1)?

hard

A. [0.2, 0.2, 0.2, 0.2, 0.2]

B. [0, 1, 0, 0, 0]

C. [0.025, 0.9, 0.025, 0.025, 0.025]

D. [0.1, 0.1, 0.1, 0.1, 0.6]

Solution

1. Step 1: Recall label smoothing formula

   With smoothing ε=0.1 and K=5 classes, true class gets 1 - ε = 0.9, each of the other K-1=4 classes gets ε / (K-1) = 0.1 / 4 = 0.025.

2. Step 2: Construct target for true class index 1

   The vector is [0.025, 0.9, 0.025, 0.025, 0.025].

3. Final Answer:

   [0.025, 0.9, 0.025, 0.025, 0.025] -> Option C

4. Quick Check:

   Smoothed target sums to 1 with 0.1 smoothing [OK]

Hint: Distribute smoothing evenly, reduce true class by smoothing [OK]

Common Mistakes:

• Using one-hot vector without smoothing
• Assigning smoothing incorrectly to true class
• Making all classes equal probability