Bird
Raised Fist0

Implement Forward Pass and Backpropagation - NumPy Only

CODINGadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Implement Forward Pass and Backpropagation for a Simple Neural Network using NumPy
30 min · senior
senior

You are tasked with implementing a simple neural network from scratch using NumPy for a startup's prototype. The network should handle a binary classification problem with a single hidden layer. The dataset is small (10,000 samples) but the model must be efficient enough to train in under 5 seconds on a standard laptop. You must implement both the forward pass and backpropagation without using any deep learning libraries.

Given Info
  • The network has one hidden layer with 10 neurons.
  • Use sigmoid activation for the hidden layer and output layer.
  • The learning rate is fixed at 0.01.
  • The loss function is binary cross-entropy.
  • The dataset is balanced.
📋
Guided Steps
1/5
Step 1
Initialize Network Parameters
How will you initialize the weights and biases for the network?
💡 Hint
Consider the impact of initialization on training stability and convergence.
Key Decision
Use small random values for weights to break symmetry and zeros for biases to simplify initial calculations.
Interviewer Expectation
Mid: Initializes weights randomly and biases to zero. Senior: Explains the importance of breaking symmetry and potential issues with large initial weights.
⚠️Common Wrong Answer
Initializes all weights and biases to zero, leading to symmetry and poor learning.
Step 2
Implement Forward Pass
How will you implement the forward pass for this network?
💡 Hint
Think about how activations are computed layer by layer.
Key Decision
Use matrix multiplication for efficient computation of activations.
Interviewer Expectation
Mid: Implements forward pass using loops. Senior: Uses NumPy's vectorized operations for efficiency and explains the computational benefits.
⚠️Common Wrong Answer
Uses Python loops for each neuron, resulting in slow execution.
Step 3
Compute Loss
How will you compute the loss for the network?
💡 Hint
Consider the properties of the binary cross-entropy loss function.
Key Decision
Use binary cross-entropy to measure the difference between predicted and actual labels.
Interviewer Expectation
Mid: Computes loss correctly. Senior: Also explains why binary cross-entropy is suitable for this problem.
⚠️Common Wrong Answer
Uses mean squared error, which is less suitable for binary classification.
Step 4
Implement Backpropagation
How will you implement backpropagation to update the network parameters?
💡 Hint
Think about how gradients flow through the network.
Key Decision
Use chain rule to compute gradients and update weights and biases.
Interviewer Expectation
Mid: Implements backpropagation with correct gradient calculations. Senior: Optimizes the gradient computation using vectorized operations and explains the chain rule application.
⚠️Common Wrong Answer
Incorrectly computes gradients, leading to ineffective learning.
Step 5
Optimize Training Loop
How will you structure the training loop to ensure efficient learning?
💡 Hint
Consider the balance between computation time and convergence.
Key Decision
Use mini-batch gradient descent to balance convergence speed and computational efficiency.
Interviewer Expectation
Mid: Uses full-batch gradient descent. Senior: Implements mini-batch gradient descent and explains its advantages in terms of convergence and computational efficiency.
⚠️Common Wrong Answer
Uses full-batch gradient descent, leading to slower convergence.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correctly initializes weights and biases to break symmetry.
  • Implements forward pass using NumPy vectorized operations.
  • Computes binary cross-entropy loss correctly.
  • Implements backpropagation with correct gradient calculations.
  • Structures training loop to ensure efficient learning.
Good to Have - 30%
  • Explains the importance of weight initialization.
  • Uses mini-batch gradient descent for training.
  • Optimizes gradient computation using vectorized operations.
  • Explains the choice of binary cross-entropy for binary classification.
Bonus - 10%
  • Discusses potential numerical stability issues and solutions.
  • Explains the impact of learning rate on convergence.
  • Considers alternative activation functions and their implications.
✅
Model Answer
▼ Reveal

The ideal solution initializes weights randomly and biases to zero, implements the forward pass using NumPy's vectorized operations, computes binary cross-entropy loss, and performs backpropagation using the chain rule. The training loop is structured with mini-batch gradient descent for efficient learning.

Key Decisions
  • Random weight initialization to break symmetry and ensure effective learning.
  • Vectorized operations for forward pass to leverage NumPy's computational efficiency.
  • Binary cross-entropy loss for its suitability in binary classification tasks.
  • Backpropagation using chain rule to correctly compute gradients.
  • Mini-batch gradient descent to balance convergence speed and computational efficiency.
Anti-patterns
Using zero initialization for weights, leading to symmetry and ineffective learning.
Implementing forward pass with Python loops, resulting in slow execution.
Choosing mean squared error for loss, which is less appropriate for binary classification.
Incorrect gradient computation during backpropagation, hindering learning.

Practice

(1/5)
1. What is the time complexity of computing the binary cross-entropy loss for a logistic regression model with n samples and m features?
easy
A. O(1)
B. O(n)
C. O(m)
D. O(nm)

Solution

  1. Step 1: Identify operations involved

    Binary cross-entropy involves computing predictions for each sample, which requires a dot product of weights and features.

  2. Step 2: Calculate complexity

    Each dot product is O(m), and there are n samples, leading to O(nm) complexity.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dot product per sample, n samples -> O(nm) [OK]
Hint: Dot product per sample, n samples -> O(nm) [OK]
Common Mistakes:
2. What is the time complexity of generating a sinusoidal positional encoding matrix of size position x d_model?
easy
A. O(d_model^2)
B. O(position^2)
C. O(position * d_model)
D. O(1)

Solution

  1. Step 1: Analyze the operations

    The code iterates over each position and each dimension, performing constant-time operations.

  2. Step 2: Eliminate wrong options

    O(position^2) is incorrect because the complexity is not quadratic in position. O(d_model^2) is incorrect because the complexity is not quadratic in d_model. O(1) is incorrect because the complexity is not constant.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Iterate over position and dimension. [OK]
Hint: Iterate over position and dimension. [OK]
Common Mistakes:
3. 
import numpy as np

def compute_confusion_matrix(y_true, y_pred):
    cm = np.zeros((2, 2), dtype=int)
    for t, p in zip(y_true, y_pred):
        cm[t][p] += 1
    return cm

# Test case
y_true = [0, 1, 0, 1, 0, 1, 1]
y_pred = [0, 0, 0, 1, 1, 1, 1]

cm = compute_confusion_matrix(y_true, y_pred)
print(cm)
Why might this implementation produce incorrect results?
medium
A. It incorrectly indexes the confusion matrix.
B. It assumes binary classification, which may not hold.
C. It does not handle ties in predictions.
D. It uses a mutable default argument.

Solution

  1. Step 1: Analyze indexing logic

    The code uses cm[t][p] which is incorrect for confusion matrix indexing.

  2. Step 2: Identify correct logic

    Correct indexing should be cm[p][t] for predictions vs actuals.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Confusion matrix should index [pred][true]. [OK]
Hint: Confusion matrix should index [pred][true]. [OK]
Common Mistakes:
4. In a Random Forest implementation, how would you apply feature sampling to handle high-dimensional data effectively?
medium
A. Use all features for each tree to ensure accuracy
B. Use a fixed subset of features for all trees
C. Randomly select a small subset of features for each tree
D. Select features based on their correlation with the target

Solution

  1. Step 1: Understand Feature Sampling

    Feature sampling involves selecting a random subset of features for each tree to reduce dimensionality.

  2. Step 2: Evaluate Options

    Using all features (A) or a fixed subset (C) doesn't reduce dimensionality; correlation-based selection (D) isn't random.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Random subsets reduce dimensionality [OK]
Hint: Random subsets reduce dimensionality [OK]
Common Mistakes:
5. In the following implementation of scaled dot-product attention, the output is incorrect. What is the likely cause? 
import torch
import torch.nn.functional as F

queries = torch.randn(2, 3)
keys = torch.randn(2, 3)
values = torch.randn(2, 3)

scores = torch.matmul(queries, keys.T) / torch.sqrt(torch.tensor(3.0))
attention_weights = F.softmax(scores, dim=0)
output = torch.matmul(attention_weights, values)
medium
A. Incorrect scaling factor used.
B. Softmax applied on the wrong dimension.
C. Values tensor has incorrect shape.
D. Scores tensor is not normalized.

Solution

  1. Step 1: Identify the issue

    Softmax should be applied on the last dimension, not the first.

  2. Step 2: Eliminate other causes

    Scaling factor and tensor shapes are correct; normalization is handled by softmax.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Softmax on last dim for attention [OK]
Hint: Softmax on last dim for attention [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard