Bird
Raised Fist0

Implement Batch Normalization - Forward and Backward Pass

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/4
Compute Mean

Mean of input batch X is calculated as 0.5

🎯 Understanding the role of mean in normalization
mean = np.mean(X, axis=0)
Compute Variance

Variance of input batch X is calculated as 0.25

🎯 Variance helps in scaling the input to unit variance
variance = np.var(X, axis=0)
Normalize

Input X is normalized to have zero mean and unit variance

🎯 Normalization is crucial for stable training
X_norm = (X - mean) / np.sqrt(variance + epsilon)
Scale and Shift

Normalized input is scaled by gamma and shifted by beta

🎯 Learned parameters gamma and beta adjust the normalized output
out = gamma * X_norm + beta
📊
Diagram
Diagram
┌───────────────────────────────────────────────────────────────┐
│                       Batch Normalization                     │
├───────────────────────────────────────────────────────────────┤
│ Input X  ──► [Compute Mean] ──► [Compute Variance] ──► [Norm] │
│   │                                                          │
│   └──────────────────────────────────────────────────────────┘
│   ┌──────────────────────────────────────────────────────────┐
│   │                                                          │
│   └──► [Scale and Shift] ──► Output Y                        │
└──────────────────────────────────────────────────────────────┘
Labels
  • Input X: The input data to be normalized
  • Compute Mean: Calculate the mean of the input batch
  • Compute Variance: Calculate the variance of the input batch
  • Norm: Normalize the input using the computed mean and variance
  • Scale and Shift: Apply learned parameters to scale and shift the normalized input
🔑
Key Moments
▲
Q: Why is the mean subtracted from the input during normalization?
A: Subtracting the mean centers the data around zero, which helps in stabilizing the learning process.
Q: What is the purpose of adding epsilon in the normalization step?
A: Epsilon is added to prevent division by zero when normalizing the input.
Q: How do gamma and beta influence the output of batch normalization?
A: Gamma scales the normalized input, and beta shifts it, allowing the model to learn the optimal scale and shift.
Trace Summary

This diagram and trace explain the forward pass of batch normalization. The process begins with computing the mean and variance of the input batch, followed by normalizing the input to have zero mean and unit variance. The normalized input is then scaled and shifted using learned parameters gamma and beta. This technique stabilizes the learning process by ensuring consistent input distributions across layers, which is crucial for deep learning models.

Primary Focus

✓ Batch Normalization algorithm implementation

Practice

(1/5)
1. What is the time complexity of computing the numerical gradient for a function with n parameters using central difference?
easy
A. O(1)
B. O(n^2)
C. O(n log n)
D. O(n)

Solution

  1. Step 1: Analyze the loop structure

    The numerical gradient computation involves a loop over n parameters, each requiring constant time operations.

  2. Step 2: Determine the complexity

    Each parameter requires a constant number of operations, leading to O(n) complexity.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Loop over n parameters, each constant time. [OK]
Hint: Loop over n parameters, each constant time. [OK]
Common Mistakes:
2. 
import numpy as np

def iou(box1, box2):
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])
    intersection = max(0, x2 - x1) * max(0, y2 - y1)
    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
    union = area1 + area2 - intersection
    return intersection / union

box1 = [1, 1, 3, 3]
box2 = [2, 2, 4, 4]
print(iou(box1, box2))
easy
A. 0.5
B. 0.25
C. 0.14285714285714285
D. 0.3333333333333333

Solution

  1. Step 1: Calculate intersection and union areas

    Intersection area is 1, union area is 7.

  2. Step 2: Compute IoU

    IoU = 1/7 = 0.14285714285714285.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Intersection over Union formula [OK]
Hint: Intersection over Union formula [OK]
Common Mistakes:
3. 
import numpy as np
from scipy.spatial import distance

X_train = np.array([[1, 2], [3, 4], [5, 6]])
X_test = np.array([[1, 2]])

# Calculate distances
D = distance.cdist(X_test, X_train, 'euclidean')

print(D)
What does the above code output?
easy
A. array([[0., 2.828, 5.656]])
B. array([[0., 2., 4.]])
C. array([[0., 2.236, 4.472]])
D. array([[0., 2.828, 4.472]])

Solution

  1. Step 1: Understand the function

    The cdist function computes the distance between each pair of the two collections of inputs.

  2. Step 2: Calculate the distances

    The Euclidean distance from [1, 2] to each point in X_train is calculated as [0., 2.828, 4.472].

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Euclidean distance formula: √((x2-x1)² + (y2-y1)²) [OK]
Hint: Euclidean distance formula: √((x2-x1)² + (y2-y1)²) [OK]
Common Mistakes:
4. What does the following Python code snippet output when implementing linear regression using gradient descent?
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])
weights = np.zeros(X.shape[1])
learning_rate = 0.01

for _ in range(1):
    predictions = X.dot(weights)
    errors = predictions - y
    gradient = X.T.dot(errors) / len(y)
    weights -= learning_rate * gradient

print(weights)
easy
A. [0.01, 0.02]
B. [0.1, 0.2]
C. [0.03, 0.04]
D. [0.05, 0.06]

Solution

  1. Step 1: Calculate initial predictions and errors

    Initial predictions are [0, 0, 0] since weights are zeros. Errors are [-1, -2, -3].

  2. Step 2: Compute gradient and update weights

    Gradient is [-3, -6]. Updated weights are [0.01, 0.02].

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Initial weights zero, single update step. [OK]
Hint: Initial weights zero, single update step. [OK]
Common Mistakes:
5. How can you optimize a naive 2D convolution implementation for large images and kernels?
hard
A. Use a larger stride to reduce computation.
B. Use a smaller kernel to decrease computation.
C. Implement convolution using FFT (Fast Fourier Transform).
D. Increase the kernel size to cover more area.

Solution

  1. Step 1: Identify optimization techniques

    Using FFT can significantly reduce the computational complexity of convolution from O(N^2 * K^2) to O(N^2 log N) for large images and kernels.

  2. Step 2: Eliminate wrong options

    Use a larger stride to reduce computation. reduces output size, not computation. Increase the kernel size to cover more area. increases computation. Use a smaller kernel to decrease computation. limits kernel effectiveness.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    FFT reduces convolution complexity. [OK]
Hint: FFT reduces convolution complexity. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard