Bird
Raised Fist0

Implement Gradient Checking - Numerical vs Analytical Gradients

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Numerical Gradient Checking vs Analytical Gradient Checking

When implementing a new neural network layer, would you use numerical or analytical gradient checking to verify your gradients? Justify your choice.

📊
Comparison
DimensionNumerical Gradient CheckingAnalytical Gradient Checking
Computational EfficiencyLowHigh
Ease of ImplementationHighMedium
ScalabilityLowHigh
PrecisionHighMedium
Suitability for DebuggingHighLow
✅
Verdict
Default to numerical gradient checking during the development phase for models with fewer than 1000 parameters or when debugging new architectures. Switch to analytical gradient checking for large-scale models in production where computational efficiency is critical.
💡
Key Insight
Numerical gradient checking provides a reliable verification method during development, while analytical gradients offer efficiency for large-scale production models.
⚠️
Interview Trap
❌ Wrong: Always use analytical gradients because they are faster.
This ignores the importance of verifying gradient correctness during initial model development.
✅ Use numerical gradient checking during development to ensure correctness, then switch to analytical gradients for efficiency in production.
Numerical Gradient Checking
"Provides a simple and reliable way to verify gradients without needing to derive them manually."
Weaknesses
  • Computationally expensive due to finite difference calculations for each parameter.
  • Prone to numerical instability if not implemented with care (e.g., choosing a proper epsilon).
  • Not suitable for large-scale models due to high computational cost.
Ideal When
  • When n < 1000 parameters.
  • When debugging a new model architecture.
  • When analytical gradients are complex to derive.
  • When precision is more critical than speed.
OpenAI uses numerical gradient checking during the initial development phase of new models to ensure correctness before scaling.
Analytical Gradient Checking
"Efficiently computes gradients using backpropagation, suitable for large-scale models."
Weaknesses
  • Requires correct implementation of backpropagation, which can be error-prone.
  • Difficult to derive for complex custom layers or operations.
  • Does not inherently verify the correctness of the gradient computation.
Ideal When
  • When n > 10,000 parameters.
  • When using well-tested libraries like PyTorch or TensorFlow.
  • When computational efficiency is critical.
  • When the model architecture is stable and well-understood.
Google uses analytical gradient checking in production models where computational efficiency is crucial and the architecture is well-tested.
🐍
Numerical Gradient Checking - Example
▼ Show
Python
import numpy as np

def numerical_gradient_check(f, x, epsilon=1e-5):
    grad_approx = np.zeros_like(x)
    for i in range(x.size):
        x_plus = np.copy(x)
        x_minus = np.copy(x)
        x_plus[i] += epsilon
        x_minus[i] -= epsilon
        grad_approx[i] = (f(x_plus) - f(x_minus)) / (2 * epsilon)
    return grad_approx
🐍
Analytical Gradient Checking - Example
▼ Show
Python
import torch

def analytical_gradient_check(model, inputs, targets):
    model.zero_grad()
    outputs = model(inputs)
    loss = torch.nn.functional.mse_loss(outputs, targets)
    loss.backward()
    return [param.grad for param in model.parameters()]
✅
Use Numerical Gradient Checking When
  • •When n < 1000 and debugging new architectures.
  • •When analytical gradients are too complex to derive manually.
  • •When precision is more critical than computational efficiency.
✅
Use Analytical Gradient Checking When
  • •When n > 10,000 and using established libraries.
  • •When computational efficiency is a priority.
  • •When the model architecture is stable and well-understood.
↳
Nuances
  • Numerical gradient checking is often used as a sanity check during initial model development, even if analytical gradients will be used later.
  • For very large models, numerical gradient checking may be infeasible due to time constraints, even if it provides more precise verification.
  • Analytical gradients can be incorrect if the backpropagation implementation has bugs, which numerical checking can help identify.

Practice

(1/5)
1. 
import numpy as np

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

vector1 = np.array([1, 2, 3])
vector2 = np.array([4, 5, 6])

result = cosine_similarity(vector1, vector2)
print(result)
easy
A. 0.9746318461970762
B. 0.9999999999999998
C. 1.0
D. 0.0

Solution

  1. Step 1: Calculate dot product and norms

    Dot product = 32, norm_a = 3.7416573867739413, norm_b = 8.774964387392123.

  2. Step 2: Compute cosine similarity

    Cosine similarity = 32 / (3.7416573867739413 * 8.774964387392123) = 0.9746318461970762.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dot product over product of norms. [OK]
Hint: Dot product over product of norms. [OK]
Common Mistakes:
2. What is the time complexity of applying Min-Max scaling to a dataset with n samples and m features?
easy
A. O(n * m)
B. O(m)
C. O(n^2)
D. O(n)

Solution

  1. Step 1: Understand Min-Max Scaling

    Min-Max scaling requires computing the min and max for each feature, which involves iterating over all samples for each feature.

  2. Step 2: Calculate Complexity

    For each of the m features, we iterate over n samples, resulting in O(n * m) complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Min-Max scaling is O(n * m). [OK]
Hint: Min-Max scaling is O(n * m). [OK]
Common Mistakes:
3. What does the following Python code snippet output when implementing Mini-Batch SGD with Momentum? 
import numpy as np

velocity = np.array([0.0, 0.0])
gradient = np.array([0.1, 0.2])
learning_rate = 0.01
momentum = 0.9

velocity = momentum * velocity - learning_rate * gradient
print(velocity)
easy
A. array([0.01, 0.02])
B. array([0.001, 0.002])
C. array([-0.001, -0.002])
D. array([-0.01, -0.02])

Solution

  1. Step 1: Calculate the velocity update

    The formula for velocity update is velocity = momentum * velocity - learning_rate * gradient. Given the initial velocity is zero, the update becomes velocity = -learning_rate * gradient.

  2. Step 2: Substitute the values

    Substitute learning_rate = 0.01 and gradient = [0.1, 0.2] into the formula: velocity = -0.01 * [0.1, 0.2] = [-0.001, -0.002].

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Velocity update is negative of learning rate times gradient [OK]
Hint: Velocity update is negative of learning rate times gradient [OK]
Common Mistakes:
4. In a recommendation system, why might you choose NDCG over precision at k?
medium
A. NDCG is easier to compute.
B. NDCG considers the position of relevant items.
C. NDCG is less sensitive to irrelevant items.
D. NDCG is more interpretable than precision.

Solution

  1. Step 1: Understand NDCG vs Precision

    NDCG accounts for the position of relevant items, giving higher scores to relevant items at higher ranks.

  2. Step 2: Evaluate Options

    Precision at k does not consider the order of items, only their presence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    NDCG values position. [OK]
Hint: NDCG values position. [OK]
Common Mistakes:
5. A Random Forest implementation is producing poor results. The trees seem to be too similar. What is the most likely cause?
medium
A. Insufficient feature sampling
B. High variance in the data
C. Too much data in each bootstrap sample
D. Too many trees in the forest

Solution

  1. Step 1: Identify the Cause of Similar Trees

    Similar trees often result from insufficient randomness in feature selection.

  2. Step 2: Eliminate Other Causes

    Too many trees (A) or high variance (D) don't cause similarity; too much data (C) affects overfitting, not similarity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feature sampling adds diversity [OK]
Hint: Feature sampling adds diversity [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard