Batch normalization (nn.BatchNorm) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of batch normalization in neural networks?

Batch normalization helps to stabilize and speed up training by normalizing the inputs of each layer, keeping their mean close to 0 and variance close to 1.

intermediate

How does nn.BatchNorm1d differ from nn.BatchNorm2d in PyTorch?

nn.BatchNorm1d is used for 2D inputs like (batch_size, features), often in fully connected layers, while nn.BatchNorm2d is for 4D inputs like (batch_size, channels, height, width), used in convolutional layers.

beginner

What are the two main parameters learned by batch normalization layers?

Batch normalization layers learn two parameters: gamma (scale) and beta (shift), which allow the network to adjust normalized outputs if needed.

intermediate

Why is batch normalization usually turned off during model evaluation?

During evaluation, batch normalization uses running estimates of mean and variance instead of batch statistics to ensure consistent outputs for single samples.

beginner

Show a simple PyTorch example of applying nn.BatchNorm1d to a tensor of shape (batch_size=4, features=3).

import torch
import torch.nn as nn

x = torch.randn(4, 3)  # random input
batch_norm = nn.BatchNorm1d(3)  # 3 features
output = batch_norm(x)
print(output)

# This normalizes each feature across the batch.

What does batch normalization normalize in a neural network?

A The loss function

B The outputs of the entire network

C The weights of the network

D The inputs to each layer

Batch normalization normalizes the inputs to each layer to have a stable mean and variance.

Which PyTorch class is used for batch normalization on 2D convolutional outputs?

A nn.BatchNorm1d

B nn.LayerNorm

C nn.BatchNorm2d

D nn.BatchNorm3d

nn.BatchNorm2d is designed for 4D tensors from convolutional layers with shape (batch, channels, height, width).

What are the learnable parameters in a batch normalization layer?

A Gamma (scale) and beta (shift)

B Weights and biases

C Mean and variance

D Learning rate and momentum

Batch normalization learns gamma and beta to scale and shift the normalized output.

Why do we use running mean and variance during evaluation in batch normalization?

A To avoid using batch statistics which may vary

B To speed up training

C To increase model complexity

D To reduce model size

Using running mean and variance ensures consistent normalization when batch size is 1 or during inference.

What happens if you forget to call model.eval() before evaluation when using batch normalization?

A Model runs faster

B Batch normalization uses batch statistics, causing inconsistent outputs

C Weights stop updating

D Nothing changes

Without model.eval(), batch normalization uses batch stats, which can cause unstable or incorrect predictions.

Explain in simple terms how batch normalization helps neural networks learn better.

Think about how keeping data stable helps learning.

Describe the difference between training and evaluation modes for batch normalization in PyTorch.

Consider what changes when you switch from training to testing.