Bird
Raised Fist0

Implement Forward Pass and Backpropagation - NumPy Only

CODINGadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Implement Forward Pass and Backpropagation - NumPy Only
Master the core of neural networks by implementing forward and backward passes using only NumPy.
codingadvancedFreq: very_highDeep Learning Coding
Prerequisites:Backpropagation - Chain RuleGradient FlowActivation Functions
📝
Problem Statement

Implement the forward pass and backpropagation for a simple neural network with one hidden layer using only NumPy. The network should use sigmoid activation for the hidden layer and mean squared error for the loss. Inputs are the feature matrix X (shape: m x n), target vector y (shape: m), and network parameters (weights and biases). Output the loss and gradients with respect to the parameters.

I/O Specification
Input:  X: np.ndarray, y: np.ndarray, params: dict
Output: loss: float, grads: dict
Example
{
  "input": "X = np.array([[0.5, 0.2], [0.9, 0.7]]), y = np.array([1, 0]), params = {'W1': np.random.rand(2, 3), 'b1': np.random.rand(3), 'W2': np.random.rand(3, 1), 'b2': np.random.rand(1)}",
  "expected_output": "loss = 0.25, grads = {'dW1': ..., 'db1': ..., 'dW2': ..., 'db2': ...}"
}
🎯
Interview Angles
  • Understanding of backpropagation and the chain rule in neural networks.
  • Ability to implement neural network operations using only NumPy, demonstrating deep understanding of matrix operations.
  • Handling edge cases and ensuring numerical stability in neural network computations.
  • Optimizing neural network computations by leveraging vectorized operations.
🔵
Naive Approach

The naive approach involves manually computing the forward pass by iterating over each layer, applying the activation function, and then calculating the loss. For backpropagation, compute gradients layer by layer using the chain rule, iterating over each parameter.

Time: O(m * n * h + h * o) - m samples, n features, h hidden units, o output units, Space: O(m * h + h * o) - storing activations and gradients
Why Insufficient
The naive approach involves explicit loops over layers and parameters, leading to inefficient computation especially for large networks or datasets.
🐍
Naive Code
Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    return sigmoid(z) * (1 - sigmoid(z))

def forward_pass(X, params):
    W1, b1, W2, b2 = params['W1'], params['b1'], params['W2'], params['b2']
    Z1 = np.dot(X, W1) + b1
    A1 = sigmoid(Z1)
    Z2 = np.dot(A1, W2) + b2
    A2 = Z2  # Linear activation for output layer
    return A1, A2

def compute_loss(A2, y):
    return np.mean((A2 - y) ** 2)

def backward_pass(X, y, A1, A2, params):
    m = X.shape[0]
    W1, b1, W2, b2 = params['W1'], params['b1'], params['W2'], params['b2']
    dZ2 = 2 * (A2 - y) / m
    dW2 = np.dot(A1.T, dZ2)
    db2 = np.sum(dZ2, axis=0)
    dA1 = np.dot(dZ2, W2.T)
    dZ1 = dA1 * sigmoid_derivative(np.dot(X, W1) + b1)
    dW1 = np.dot(X.T, dZ1)
    db1 = np.sum(dZ1, axis=0)
    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def neural_network(X, y, params):
    A1, A2 = forward_pass(X, params)
    loss = compute_loss(A2, y)
    grads = backward_pass(X, y, A1, A2, params)
    return loss, grads

# Driver example
X = np.array([[0.5, 0.2], [0.9, 0.7]])
y = np.array([1, 0])
params = {'W1': np.random.rand(2, 3), 'b1': np.random.rand(3), 'W2': np.random.rand(3, 1), 'b2': np.random.rand(1)}
loss, grads = neural_network(X, y, params)
print('Loss:', loss)
print('Gradients:', grads)
💡
Key Insight
Utilize NumPy's vectorized operations to perform matrix multiplications and element-wise operations efficiently, eliminating explicit loops over samples and parameters.
🐍
Optimized Code
Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    return sigmoid(z) * (1 - sigmoid(z))

def forward_pass(X, params):
    W1, b1, W2, b2 = params['W1'], params['b1'], params['W2'], params['b2']
    Z1 = np.dot(X, W1) + b1
    A1 = sigmoid(Z1)
    Z2 = np.dot(A1, W2) + b2
    A2 = Z2  # Linear activation for output layer
    return A1, A2

def compute_loss(A2, y):
    return np.mean((A2 - y) ** 2)

def backward_pass(X, y, A1, A2, params):
    m = X.shape[0]
    W1, b1, W2, b2 = params['W1'], params['b1'], params['W2'], params['b2']
    dZ2 = 2 * (A2 - y) / m
    dW2 = np.dot(A1.T, dZ2)
    db2 = np.sum(dZ2, axis=0)
    dA1 = np.dot(dZ2, W2.T)
    dZ1 = dA1 * sigmoid_derivative(np.dot(X, W1) + b1)
    dW1 = np.dot(X.T, dZ1)
    db1 = np.sum(dZ1, axis=0)
    grads = {'dW1': dW1, 'db1': db1, 'dW2': dW2, 'db2': db2}
    return grads

def neural_network(X, y, params):
    A1, A2 = forward_pass(X, params)
    loss = compute_loss(A2, y)
    grads = backward_pass(X, y, A1, A2, params)
    return loss, grads

# Driver example
X = np.array([[0.5, 0.2], [0.9, 0.7]])
y = np.array([1, 0])
params = {'W1': np.random.rand(2, 3), 'b1': np.random.rand(3), 'W2': np.random.rand(3, 1), 'b2': np.random.rand(1)}
loss, grads = neural_network(X, y, params)
print('Loss:', loss)
print('Gradients:', grads)
🔍
Line-by-Line
Z1 = np.dot(X, W1) + b1
Compute the linear combination of inputs and weights for the hidden layer - vectorized over all samples.
A1 = sigmoid(Z1)
Apply the sigmoid activation function element-wise to the hidden layer's linear output.
Z2 = np.dot(A1, W2) + b2
Compute the linear combination for the output layer - vectorized over all samples.
dZ2 = 2 * (A2 - y) / m
Calculate the gradient of the loss with respect to the output layer's linear output.
dW2 = np.dot(A1.T, dZ2)
Compute the gradient of the loss with respect to W2 using the chain rule - vectorized matrix multiplication.
db2 = np.sum(dZ2, axis=0)
Sum the gradients over all samples to get the gradient with respect to b2.
dA1 = np.dot(dZ2, W2.T)
Backpropagate the gradient to the hidden layer's activation.
dZ1 = dA1 * sigmoid_derivative(np.dot(X, W1) + b1)
Apply the chain rule to get the gradient with respect to the hidden layer's linear output.
⚡
Complexity
Time: O(m * n * h + h * o) - matrix multiplications dominate, each operation is vectorized, Space: O(m * h + h * o) - storing activations and gradients
Output
Loss: 0.25
Gradients: {'dW1': ..., 'db1': ..., 'dW2': ..., 'db2': ...}
🔲
Edge Cases
Empty Input
The function should handle empty input arrays gracefully without errors.
m = X.shape[0] - ensures operations are safe even if m = 0
Single Sample
The function should correctly compute the loss and gradients for a single input sample.
np.dot(X, W1) - handles single sample as a special case of matrix multiplication
⚠️
Common Mistakes
Incorrectly computing the derivative of the sigmoid function.
Symptom: Gradients are incorrect, leading to poor convergence during training.
Forgetting to divide by the number of samples when computing the gradient of the loss.
Symptom: Gradients are too large, causing instability in training.
Incorrectly handling the bias terms during backpropagation.
Symptom: Bias gradients are zero or incorrect, affecting model performance.
Using loops instead of vectorized operations for matrix multiplications.
Symptom: Code runs significantly slower, especially on larger datasets.

Practice

(1/5)
1. What is the time complexity of performing K-Fold cross-validation on a dataset with n samples and k folds?
easy
A. O(n^2)
B. O(n)
C. O(kn)
D. O(k^2n)

Solution

  1. Step 1: Analyze K-Fold process

    Each fold requires training on (k-1)/k of the data and testing on 1/k.

  2. Step 2: Eliminate wrong options

    O(n) ignores the k factor. O(n^2) and D overestimate complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    K-Fold complexity is O(kn). [OK]
Hint: K-Fold complexity is O(kn). [OK]
Common Mistakes:
2. 
import numpy as np

def numerical_gradient(f, x):
    h = 1e-4
    grad = np.zeros_like(x)
    for i in range(x.size):
        x[i] += h
        fxh1 = f(x)
        x[i] -= 2 * h
        fxh2 = f(x)
        grad[i] = (fxh1 - fxh2) / (2 * h)
        x[i] += h
    return grad

# Test function
f = lambda x: x[0]**2 + x[1]**2
x = np.array([3.0, 4.0])
print(numerical_gradient(f, x))
easy
A. [6.0, 8.0]
B. [6.0001, 8.0001]
C. [5.9999, 7.9999]
D. [0.0, 0.0]

Solution

  1. Step 1: Understand the numerical gradient calculation

    The function calculates the gradient using central difference approximation.

  2. Step 2: Calculate the gradient for each element

    For x = [3.0, 4.0], the gradient is [6.0, 8.0] because the derivative of x^2 is 2x.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Central difference gives exact gradient for quadratic. [OK]
Hint: Central difference gives exact gradient for quadratic. [OK]
Common Mistakes:
3. You implemented a 2D convolution but the output is all zeros. What is the most likely cause?
medium
A. The kernel is not normalized.
B. The kernel is filled with zeros.
C. The image is padded incorrectly.
D. The stride is set too high.

Solution

  1. Step 1: Identify the zero output cause

    If the kernel is filled with zeros, every convolution operation will result in zero, regardless of the image content.

  2. Step 2: Eliminate wrong options

    The kernel is not normalized. affects scaling, not zero output. The image is padded incorrectly. affects boundary handling, not zero output. The stride is set too high. affects output size, not zero output.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Zero kernel leads to zero output. [OK]
Hint: Zero kernel leads to zero output. [OK]
Common Mistakes:
4. You implemented stratified K-Fold cross-validation but notice class imbalance in some folds. What is the most likely cause?
medium
A. Incorrect stratification parameter
B. Data shuffling before splitting
C. Incorrect number of splits
D. Too many folds for the dataset size

Solution

  1. Step 1: Identify stratification issue

    Stratification ensures class distribution is preserved in each fold.

  2. Step 2: Eliminate wrong options

    Incorrect number of splits and D relate to fold count, not class distribution. Data shuffling before splitting affects randomness, not stratification.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Check stratification parameter for class balance. [OK]
Hint: Check stratification parameter for class balance. [OK]
Common Mistakes:
5. How can you optimize the implementation of K-Fold cross-validation to reduce computational overhead?
hard
A. Use fewer folds
B. Increase the number of epochs
C. Use a smaller dataset
D. Parallelize the training process

Solution

  1. Step 1: Identify optimization strategies

    Parallelizing the training process can significantly reduce computation time.

  2. Step 2: Eliminate wrong options

    Use fewer folds reduces validation quality. Use a smaller dataset affects model performance. Increase the number of epochs increases computation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Parallelize to optimize K-Fold. [OK]
Hint: Parallelize to optimize K-Fold. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard