Bird
Raised Fist0

Batch Normalization vs Layer Normalization - Why and When

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/6
Batch Normalization - Input

Mini-batch of size 32 is fed into the network.

🎯 Batch Normalization requires a mini-batch to compute statistics.
batch_mean = np.mean(batch, axis=0)
batch_var = np.var(batch, axis=0)
Batch Normalization - Normalize

Features are normalized using batch mean and variance.

🎯 Normalization helps in stabilizing the learning process.
normalized_batch = (batch - batch_mean) / np.sqrt(batch_var + epsilon)
Batch Normalization - Scale & Shift

Normalized features are scaled and shifted.

🎯 Scale and shift parameters are learnable, allowing flexibility.
scaled_batch = gamma * normalized_batch + beta
Layer Normalization - Input

Single data instance is processed.

🎯 Layer Normalization does not depend on batch size.
layer_mean = np.mean(instance)
layer_var = np.var(instance)
Layer Normalization - Normalize

Normalization is applied across the layer.

🎯 Layer Normalization is beneficial for recurrent networks.
normalized_instance = (instance - layer_mean) / np.sqrt(layer_var + epsilon)
Layer Normalization - Scale & Shift

Normalized instance is scaled and shifted.

🎯 Similar to Batch Norm, but applied per instance.
scaled_instance = gamma * normalized_instance + beta
📊
Diagram
Diagram
┌───────────────────────────────┐
│ Batch Normalization           │
├───────────────────────────────┤
│ Input: Mini-batch of data     │
│ Step 1: Compute mean & var    │
│ Step 2: Normalize each feature│
│ Step 3: Scale & shift         │
└───────────────────────────────┘
          │
          ▼
┌───────────────────────────────┐
│ Layer Normalization           │
├───────────────────────────────┤
│ Input: Single data instance   │
│ Step 1: Compute mean & var    │
│ Step 2: Normalize across layer│
│ Step 3: Scale & shift         │
└───────────────────────────────┘
Labels
  • Batch Normalization: Operates on mini-batches, normalizing each feature.
  • Layer Normalization: Operates on individual instances, normalizing across the layer.
  • Mean & Variance: Calculated for normalization in both methods.
  • Scale & Shift: Applied after normalization to maintain model capacity.
🔑
Key Moments
▲
Q: Why does Batch Normalization require a mini-batch?
A: Batch Normalization computes mean and variance across a mini-batch to normalize features, which helps in reducing internal covariate shift.
Q: How does Layer Normalization differ in its application compared to Batch Normalization?
A: Layer Normalization normalizes across the features of a single instance, making it independent of batch size and suitable for recurrent networks.
Q: What is the role of the scale and shift parameters in normalization?
A: The scale (gamma) and shift (beta) parameters are learnable and allow the model to maintain its capacity to represent complex functions after normalization.
Trace Summary

The diagram and trace steps illustrate the mechanisms of Batch Normalization and Layer Normalization. Batch Normalization operates on mini-batches, normalizing each feature using batch statistics, which helps stabilize learning by reducing internal covariate shift. Layer Normalization, on the other hand, normalizes across the features of a single instance, making it independent of batch size and particularly useful for recurrent networks. Both methods apply learnable scale and shift parameters after normalization to retain model expressiveness.

Primary Focus

✓ Understanding the differences and applications of Batch Normalization and Layer Normalization in deep learning models.

Practice

(1/5)
1. 
import numpy as np
from sklearn.preprocessing import StandardScaler

X = np.array([[1, 2], [3, 4], [5, 6]])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled.mean(axis=0))
What does this code output?
easy
A. [1. 1.]
B. [0.5 0.5]
C. [0. 0.]
D. [0. 1.]

Solution

  1. Step 1: Understand standardization.

    StandardScaler standardizes features by removing the mean and scaling to unit variance.

  2. Step 2: Calculate the mean of standardized data.

    After standardization, the mean of each feature should be 0.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    StandardScaler centers data to mean 0. [OK]
Hint: StandardScaler centers data to mean 0. [OK]
Common Mistakes:
2. In the context of framing a business problem as an ML problem, which of the following is the most crucial first step?
easy
A. Selecting the right algorithm
B. Collecting as much data as possible
C. Ensuring high computational resources
D. Defining the business objective clearly

Solution

  1. Step 1: Identify the core requirement

    The first step in framing a business problem as an ML problem is to clearly define the business objective. Without a clear objective, the ML model cannot be aligned with business goals.

  2. Step 2: Eliminate wrong options

    Selecting the right algorithm is incorrect because algorithm selection comes after defining the problem. Collecting as much data as possible is incorrect because data collection should be guided by the problem definition. Ensuring high computational resources is incorrect because computational resources are a consideration after the problem is defined.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Start with a clear business goal. [OK]
Hint: Start with a clear business goal. [OK]
Common Mistakes:
3. When choosing between a Graph Convolutional Network (GCN) and a Graph Attention Network (GAT) for a task with highly heterogeneous node importance, which is more appropriate and why?
medium
A. GCN, because it is simpler and faster for all graph tasks.
B. GAT, because it can assign different importance to different nodes.
C. GCN, because it uses attention mechanisms to focus on important nodes.
D. GAT, because it is always better than GCN in every scenario.

Solution

  1. Step 1: Identify the task requirement

    Highly heterogeneous node importance suggests the need for attention mechanisms.

  2. Step 2: Compare GCN and GAT

    GAT uses attention to weigh node importance, unlike GCN.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Use GAT for node importance heterogeneity. [check]
Hint: Use GAT for node importance heterogeneity. [check]
Common Mistakes:
4. When choosing between online and offline data collection for a recommendation system, which factor is most critical for deciding to use online data collection?
medium
A. The need for real-time user feedback integration.
B. The availability of large historical datasets.
C. The requirement for high computational resources.
D. The simplicity of the data processing pipeline.

Solution

  1. Step 1: Identify the key factor for online data collection

    Online data collection is crucial when real-time feedback is needed to update recommendations dynamically.

  2. Step 2: Eliminate factors not specific to online collection

    Options B, C, and D do not specifically require online data collection; they can be addressed with offline methods.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs = online collection. [OK]
Hint: Real-time needs = online collection. [OK]
Common Mistakes:
5. For a large streaming dataset where you need to maintain a representative sample, which sampling strategy is most appropriate and why?
medium
A. Stratified Sampling to ensure class balance.
B. Reservoir Sampling to handle streaming data efficiently.
C. Random Sampling because it's simple and fast.
D. Weighted Sampling to prioritize important data.

Solution

  1. Step 1: Identify the need for streaming data handling.

    Streaming data requires a strategy that can update samples dynamically.

  2. Step 2: Eliminate non-streaming strategies.

    Random (A) and Stratified (B) don't handle streaming efficiently. Weighted (D) is for prioritization, not streaming.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Reservoir = Streaming data handling [[check]]
Hint: Reservoir = Streaming data handling [[check]]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard