Label smoothing in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of label smoothing in classification?

Label smoothing is a technique used during training classification models. What is its main purpose?

A To reduce overconfidence of the model by softening the target labels

B To increase the learning rate dynamically during training

C To add noise to the input data for better generalization

D To convert multi-class problems into multiple binary problems

💡 Hint

Think about how label smoothing changes the target labels.

Label smoothing replaces hard 0 and 1 labels with softer values to prevent the model from becoming too confident, which helps generalization.

❓ Predict Output

intermediate

What is the output tensor after applying label smoothing?

Given the following PyTorch code applying label smoothing to a batch of 3 samples with 4 classes, what is the smoothed target tensor?

PyTorch

import torch

def smooth_labels(labels, smoothing=0.1, num_classes=4):
    with torch.no_grad():
        smooth_value = smoothing / num_classes
        one_hot = torch.zeros(labels.size(0), num_classes)
        one_hot.scatter_(1, labels.unsqueeze(1), 1.0)
        return one_hot * (1 - smoothing) + smooth_value

labels = torch.tensor([0, 2, 3])
smoothed = smooth_labels(labels)
print(smoothed)

A [[1, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

B [[0.9, 0.1, 0.1, 0.1], [0.1, 0.1, 0.9, 0.1], [0.1, 0.1, 0.1, 0.9]]

C [[0.925, 0.025, 0.025, 0.025], [0.025, 0.025, 0.925, 0.025], [0.025, 0.025, 0.025, 0.925]]

D [[0.9, 0.0333, 0.0333, 0.0333], [0.0333, 0.0333, 0.9, 0.0333], [0.0333, 0.0333, 0.0333, 0.9]]

💡 Hint

Calculate smoothing value as smoothing/num_classes and add it to all classes after scaling the one-hot by (1-smoothing).

With smoothing=0.1 and 4 classes, smoothing value is 0.1/4=0.025. The one-hot is scaled by (1-0.1)=0.9, then 0.025 is added to all, resulting in 0.925 for target and 0.025 for others.

❓ Hyperparameter

advanced

Choosing the label smoothing factor

Which label smoothing factor is most likely to cause the model to underfit?

A 0.5

B 0.05

C 0.0 (no smoothing)

D 0.1

💡 Hint

Think about what happens if the smoothing factor is very large.

A very high smoothing factor like 0.5 makes the target labels too soft, reducing the model's ability to learn clear distinctions, leading to underfitting.

❓ Metrics

advanced

Effect of label smoothing on training loss

When using label smoothing during training, how does the training loss typically behave compared to training without label smoothing?

A Training loss fluctuates randomly and unpredictably

B Training loss is lower because the model fits the smoothed labels better

C Training loss is the same because label smoothing does not affect loss values

D Training loss is higher because the model is penalized for being too confident

💡 Hint

Consider how smoothing changes the target labels and model confidence.

Label smoothing prevents the model from assigning full probability to the correct class, so the loss is higher since the model is penalized for overconfidence.

🔧 Debug

expert

Identify the error in this label smoothing implementation

What error will this PyTorch code raise when applying label smoothing?

PyTorch

import torch

def label_smooth(targets, smoothing=0.1, num_classes=5):
    smooth_val = smoothing / num_classes
    one_hot = torch.zeros(targets.size(0), num_classes)
    one_hot.scatter_(1, targets.unsqueeze(1), 1.0)
    return one_hot * (1 - smoothing) + smooth_val

targets = torch.tensor([1, 3, 4])
smoothed = label_smooth(targets)
print(smoothed)

A RuntimeError due to scatter_ index out of range

B No error, outputs smoothed labels correctly

C TypeError because smoothing is divided by num_classes instead of num_classes - 1

D ValueError because targets tensor shape is incompatible

💡 Hint

Check the smoothing value calculation and scatter_ usage carefully.

The code correctly implements a common variant of label smoothing using smoothing/num_classes, and scatter_ is used properly with valid indices (0 to 4 for num_classes=5). No error occurs.