Batch normalization (nn.BatchNorm) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of BatchNorm1d on a simple tensor

What is the output tensor after applying nn.BatchNorm1d to the input tensor below during training mode?

PyTorch

import torch
import torch.nn as nn

batch_norm = nn.BatchNorm1d(num_features=3)
input_tensor = torch.tensor([[1.0, 2.0, 3.0],
                             [4.0, 5.0, 6.0]], dtype=torch.float32)
output = batch_norm(input_tensor)
print(output)

A [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]

B [[ -1.2247, -1.2247, -1.2247], [1.2247, 1.2247, 1.2247]]

C [[ -1.0, -1.0, -1.0], [1.0, 1.0, 1.0]]

D [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]

💡 Hint

BatchNorm normalizes each feature across the batch to have mean 0 and variance 1 during training.

BatchNorm1d normalizes each feature dimension independently. For the input, each feature's mean and variance are computed across the batch, then each value is normalized by subtracting the mean and dividing by the standard deviation. The output values are approximately ±1.2247 due to standard deviation sqrt(4.5).

🧠 Conceptual

intermediate

Purpose of running mean and variance in BatchNorm

What is the main purpose of the running mean and running variance in nn.BatchNorm layers during training and evaluation?

A They are used to initialize the weights of the BatchNorm layer.

B They are only used to compute gradients during backpropagation.

C They keep a moving average of mean and variance to use during evaluation mode.

D They store the mean and variance of the current batch only.

💡 Hint

Think about how BatchNorm behaves differently during training and evaluation.

During training, BatchNorm uses batch statistics to normalize inputs. It also updates running mean and variance as moving averages. During evaluation, it uses these running statistics to normalize inputs instead of batch statistics.

❓ Hyperparameter

advanced

Effect of momentum parameter in nn.BatchNorm

In nn.BatchNorm, what effect does increasing the momentum parameter have on the running mean and variance?

A It makes the running statistics update faster, relying more on the current batch.

B It makes the running statistics update more slowly, relying more on past values.

C It disables the running statistics updates entirely.

D It changes the learning rate of the BatchNorm layer.

💡 Hint

Momentum controls how much weight is given to new batch statistics versus old running statistics.

Higher momentum means the running mean and variance incorporate more of the current batch's statistics, updating faster. Lower momentum means slower updates, relying more on past values.

🔧 Debug

advanced

Identifying error in BatchNorm usage

What error will occur when running the following code snippet and why? import torch import torch.nn as nn batch_norm = nn.BatchNorm2d(num_features=3) input_tensor = torch.randn(10, 5, 32, 32) output = batch_norm(input_tensor)

PyTorch

import torch
import torch.nn as nn

batch_norm = nn.BatchNorm2d(num_features=3)
input_tensor = torch.randn(10, 5, 32, 32)
output = batch_norm(input_tensor)

A ValueError: num_features must be equal to batch size.

B TypeError: nn.BatchNorm2d does not accept 4D tensors.

C No error, output tensor shape is (10, 3, 32, 32).

D RuntimeError: Expected 3 channels but got 5 channels in input tensor.

💡 Hint

Check the channel dimension of the input tensor and the num_features parameter.

nn.BatchNorm2d expects the input tensor to have shape (batch_size, num_features, height, width). Here, num_features=3 but input has 5 channels, causing a runtime error.

❓ Model Choice

expert

Choosing BatchNorm variant for sequence data

You are building a neural network to process variable-length sequences of word embeddings with shape (batch_size, sequence_length, embedding_dim). Which BatchNorm variant is most appropriate to normalize the embeddings across the batch and sequence length?

A nn.BatchNorm1d applied on embedding_dim dimension after reshaping to (batch_size*sequence_length, embedding_dim).

B nn.BatchNorm2d applied directly on (batch_size, sequence_length, embedding_dim) tensor.

C nn.BatchNorm3d applied on the input tensor.

D No BatchNorm variant is suitable; use LayerNorm instead.

💡 Hint

Consider how BatchNorm1d expects input shape and what dimension to normalize.

nn.BatchNorm1d expects input of shape (batch_size, num_features) or (batch_size, num_features, length). Reshaping to (batch_size*sequence_length, embedding_dim) allows normalization over embedding features. BatchNorm2d and 3d expect 4D or 5D tensors, not suitable here. LayerNorm is an alternative but not a BatchNorm variant.