Label smoothing in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a label smoothing loss using PyTorch.

PyTorch

import torch.nn as nn

criterion = nn.CrossEntropyLoss(label_smoothing=[1])

Drag options to blanks, or click blank then click option'

A0.1

B1.0

C0.0

D-0.1

Attempts:

3 left

💡 Hint

Common Mistakes

Using 1.0 which removes all confidence.

Using negative values which are invalid.

Label smoothing is usually a small positive value like 0.1 to soften the labels.

2fill in blank

medium

Complete the code to apply label smoothing in a training loop.

PyTorch

outputs = model(inputs)
loss = criterion(outputs, [1])

Drag options to blanks, or click blank then click option'

Asmoothed_labels

Braw_labels

Cinputs

Doutputs

Attempts:

3 left

💡 Hint

Common Mistakes

Passing smoothed labels which causes errors.

Passing model outputs as labels.

The loss function expects the original raw labels (class indices), not smoothed labels.

3fill in blank

hard

Fix the error in the label smoothing parameter to avoid invalid values.

PyTorch

criterion = nn.CrossEntropyLoss(label_smoothing=[1])

Drag options to blanks, or click blank then click option'

A0.0

B-0.05

C1.5

D0.15

Attempts:

3 left

💡 Hint

Common Mistakes

Using negative or greater than 1 values causes runtime errors.

Label smoothing must be between 0 and 1. 0.15 is valid, others are invalid.

4fill in blank

hard

Fill both blanks to create a smoothed label tensor for a batch of size 3 and 5 classes.

PyTorch

import torch

batch_size = 3
num_classes = 5
smoothing = [1]
labels = torch.tensor([0, 2, 4])
smoothed_labels = torch.full((batch_size, num_classes), [2])
smoothed_labels.scatter_(1, labels.unsqueeze(1), 1 - smoothing)

Drag options to blanks, or click blank then click option'

A0.1

B0.9

C0.025

D0.5

Attempts:

3 left

💡 Hint

Common Mistakes

Using 0.9 as smoothing which is too high.

Using 0.5 as off-target value which is incorrect.

Label smoothing value is 0.1, so smoothing mass is spread as 0.025 to other classes (0.1 / (5-1)).

5fill in blank

hard

Fill all three blanks to compute the label smoothing loss manually.

PyTorch

import torch
import torch.nn.functional as F

outputs = torch.tensor([[2.0, 0.5, 0.3], [0.1, 1.0, 2.1]])
labels = torch.tensor([0, 2])
smoothing = [1]
num_classes = outputs.size(1)
with torch.no_grad():
    true_dist = torch.full_like(outputs, [2])
    true_dist.scatter_(1, labels.unsqueeze(1), [3])
log_probs = F.log_softmax(outputs, dim=1)
loss = (-true_dist * log_probs).sum(dim=1).mean()

Drag options to blanks, or click blank then click option'

A0.1

B0.05

C0.9

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up smoothing and target class values.

Not using torch.no_grad() when creating true_dist.

Smoothing is 0.1, off-target value is 0.1/(3-1)=0.05, and target class value is 0.9 (1 - smoothing).

Practice

1. What is the main purpose of label smoothing in PyTorch?

easy

A. To increase the learning rate automatically

B. To make the model less confident and improve generalization

C. To add noise to the input data

D. To reduce the size of the training dataset

Solution

1. Step 1: Understand label smoothing concept

   Label smoothing softens the target labels, making the model less confident about the exact class.

2. Step 2: Connect to model behavior

   This helps the model generalize better by not being too sure, reducing overfitting.

3. Final Answer:

   To make the model less confident and improve generalization -> Option B

4. Quick Check:

   Label smoothing = less confident model [OK]

Hint: Label smoothing reduces confidence to improve generalization [OK]

Common Mistakes:

• Thinking it changes learning rate
• Confusing with data augmentation
• Assuming it reduces dataset size

2. Which of the following is the correct way to apply label smoothing in PyTorch's CrossEntropyLoss?

easy

A. loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

B. loss_fn = torch.nn.CrossEntropyLoss(smooth_labels=0.1)

C. loss_fn = torch.nn.CrossEntropyLoss(smoothing=0.1)

D. loss_fn = torch.nn.CrossEntropyLoss(label_smooth=0.1)

Solution

1. Step 1: Recall PyTorch CrossEntropyLoss parameters

   The correct parameter name for label smoothing is exactly 'label_smoothing'.

2. Step 2: Match correct syntax

   Only loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) uses the exact parameter name and value format.

3. Final Answer:

   loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1) -> Option A

4. Quick Check:

   Parameter name is 'label_smoothing' [OK]

Hint: Use exact parameter name 'label_smoothing' in CrossEntropyLoss [OK]

Common Mistakes:

• Using incorrect parameter names like 'smooth_labels'
• Misspelling 'label_smoothing'
• Passing label smoothing outside loss function

3. Given the following code snippet, what will be the printed loss value trend when label smoothing is applied?

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.2)
logits = torch.tensor([[2.0, 0.5, 0.3]])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(round(loss.item(), 3))

medium

A. Loss will be negative

B. Loss will be zero

C. Loss will be lower than without label smoothing

D. Loss will be higher than without label smoothing

Solution

1. Step 1: Understand effect of label smoothing on loss

   Label smoothing softens the target, so the loss does not become zero even if prediction is perfect.

2. Step 2: Compare loss values

   Without smoothing, loss can be very low; with smoothing, loss is higher because targets are less certain.

3. Final Answer:

   Loss will be higher than without label smoothing -> Option D

4. Quick Check:

   Label smoothing increases loss value slightly [OK]

Hint: Label smoothing raises loss by softening targets [OK]

Common Mistakes:

• Expecting loss to be zero with smoothing
• Thinking smoothing lowers loss always
• Confusing loss sign (negative)

4. Identify the error in this PyTorch code snippet using label smoothing:

import torch
loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
logits = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[2]])
loss = loss_fn(logits, target)
print(loss.item())

medium

A. Target tensor shape should be 1D, not 2D

B. Label smoothing parameter must be an integer

C. Logits tensor should be 1D, not 2D

D. CrossEntropyLoss does not support label smoothing

Solution

1. Step 1: Check target tensor shape

   CrossEntropyLoss expects target as 1D tensor of class indices, but target is 2D here.

2. Step 2: Confirm label smoothing usage

   Label smoothing parameter is correctly used as float; logits shape is correct as batch size 1 with 3 classes.

3. Final Answer:

   Target tensor shape should be 1D, not 2D -> Option A

4. Quick Check:

   Target shape must be 1D for CrossEntropyLoss [OK]

Hint: Target tensor must be 1D class indices [OK]

Common Mistakes:

• Passing target as 2D tensor
• Using integer for label_smoothing
• Misunderstanding CrossEntropyLoss support

5. You want to train a classification model with 5 classes using label smoothing of 0.1. Which of the following target label vectors correctly applies label smoothing manually for class 2 (index 1)?

hard

A. [0.2, 0.2, 0.2, 0.2, 0.2]

B. [0, 1, 0, 0, 0]

C. [0.025, 0.9, 0.025, 0.025, 0.025]

D. [0.1, 0.1, 0.1, 0.1, 0.6]

Solution

1. Step 1: Recall label smoothing formula

   With smoothing ε=0.1 and K=5 classes, true class gets 1 - ε = 0.9, each of the other K-1=4 classes gets ε / (K-1) = 0.1 / 4 = 0.025.

2. Step 2: Construct target for true class index 1

   The vector is [0.025, 0.9, 0.025, 0.025, 0.025].

3. Final Answer:

   [0.025, 0.9, 0.025, 0.025, 0.025] -> Option C

4. Quick Check:

   Smoothed target sums to 1 with 0.1 smoothing [OK]

Hint: Distribute smoothing evenly, reduce true class by smoothing [OK]

Common Mistakes:

• Using one-hot vector without smoothing
• Assigning smoothing incorrectly to true class
• Making all classes equal probability