Bird
Raised Fist0

Implement Gradient Checking - Numerical vs Analytical Gradients

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/5
Initial Parameters

Parameters initialized to [0.5, -0.5]

🎯 Understanding initial parameter setup is crucial for gradient checking.
params = np.array([0.5, -0.5])
Analytical Gradient Calculation

Analytical gradient computed as [1.0, -1.0]

🎯 Analytical gradients are derived from the model's backpropagation.
analytical_grad = np.array([1.0, -1.0])
Numerical Gradient Calculation

Numerical gradient computed as [1.01, -0.99]

🎯 Numerical gradients use finite differences for approximation.
epsilon = 1e-4
numerical_grad = (f(params + epsilon) - f(params - epsilon)) / (2 * epsilon)
Gradient Comparison

Difference between gradients is [0.01, 0.01]

🎯 Comparing gradients ensures the analytical gradient is correct.
difference = np.linalg.norm(analytical_grad - numerical_grad)
Threshold Check

Difference is within the threshold of 1e-7

🎯 Ensuring the difference is below a threshold confirms gradient accuracy.
assert difference < 1e-7, 'Gradient check failed!'
📊
Diagram
Diagram
┌──────────────────────────────────────────────────────────────┐
│                        Gradient Checking                     │
├──────────────────────────────────────────────────────────────┤
│ Analytical Gradient: ∇f(x) = [df/dx1, df/dx2, ..., df/dxn]   │
│ Numerical Gradient:  ∇f(x) ≈ [(f(x+ε) - f(x-ε))/(2ε)]        │
│                                                              │
│ Compare: |∇f_analytical - ∇f_numerical| < threshold          │
│                                                              │
│ ε: small perturbation                                        │
└──────────────────────────────────────────────────────────────┘
Labels
  • Analytical Gradient: Derived from backpropagation
  • Numerical Gradient: Computed using finite differences
  • Comparison: Ensures gradients are similar within a threshold
  • ε: Small perturbation for numerical approximation
🔑
Key Moments
▲
Q: Why is a small perturbation ε used in numerical gradient calculation?
A: A small ε ensures that the numerical approximation is accurate without causing significant rounding errors.
Q: What does the comparison between analytical and numerical gradients reveal?
A: It reveals whether the analytical gradient is correctly implemented by checking if the difference is within a small threshold.
Q: Why is it important to ensure the difference between gradients is below a threshold?
A: Ensuring the difference is below a threshold confirms that the analytical gradient is accurate and reliable for optimization.
Trace Summary

The diagram and trace illustrate the process of gradient checking, which involves comparing analytical gradients derived from backpropagation with numerical gradients computed using finite differences. This comparison ensures the correctness of the analytical gradients. The process begins with initializing parameters, calculating both gradients, and then comparing them to ensure their difference is within a specified threshold. This verification step is crucial for confirming the accuracy of the gradient calculations, which are essential for effective model optimization.

Primary Focus

✓ Gradient Checking

Practice

(1/5)
1. 
import numpy as np

def dcg_at_k(r, k):
    r = np.asfarray(r)[:k]
    if r.size:
        return np.sum(r / np.log2(np.arange(2, r.size + 2)))
    return 0.0

relevance_scores = [3, 2, 3, 0, 1, 2]
k = 3
print(dcg_at_k(relevance_scores, k))
What does this code output?
easy
A. 5.2618595071429155
B. 4.2618595071429155
C. 3.2618595071429155
D. 6.2618595071429155

Solution

  1. Step 1: Understand DCG Calculation

    The DCG is calculated as the sum of relevance scores divided by log base 2 of their positions.

  2. Step 2: Calculate for k=3

    For k=3, the scores are [3, 2, 3]. The calculation is 3/1 + 2/log2(3) + 3/log2(4).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DCG uses log2 positions. [OK]
Hint: DCG uses log2 positions. [OK]
Common Mistakes:
2. What does the following Python code output?
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

x = np.array([2.0, 1.0, 0.1])
print(softmax(x))
easy
A. [0.659, 0.242, 0.099]
B. [0.5, 0.3, 0.2]
C. [0.665, 0.244, 0.091]
D. [0.7, 0.2, 0.1]

Solution

  1. Step 1: Calculate softmax

    Compute the exponentials of the input array after subtracting the max value for numerical stability.

  2. Step 2: Normalize

    Divide each exponential by the sum of all exponentials to get probabilities.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Softmax outputs sum to 1 [OK]
Hint: Softmax outputs sum to 1 [OK]
Common Mistakes:
3. You implemented stratified K-Fold cross-validation but notice class imbalance in some folds. What is the most likely cause?
medium
A. Incorrect stratification parameter
B. Data shuffling before splitting
C. Incorrect number of splits
D. Too many folds for the dataset size

Solution

  1. Step 1: Identify stratification issue

    Stratification ensures class distribution is preserved in each fold.

  2. Step 2: Eliminate wrong options

    Incorrect number of splits and D relate to fold count, not class distribution. Data shuffling before splitting affects randomness, not stratification.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Check stratification parameter for class balance. [OK]
Hint: Check stratification parameter for class balance. [OK]
Common Mistakes:
4. You implemented early stopping in your training loop, but the model still overfits. What is the most likely reason?
medium
A. The validation set is too small.
B. The learning rate is too high.
C. The patience parameter is too low.
D. The model architecture is too simple.

Solution

  1. Step 1: Analyze early stopping behavior

    Early stopping relies on validation performance to halt training.

  2. Step 2: Identify the root cause of overfitting

    If the validation set is too small, it may not accurately reflect generalization, leading to overfitting despite early stopping.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Small validation set can mislead early stopping. [OK]
Hint: Small validation set can mislead early stopping. [OK]
Common Mistakes:
5. In the context of implementing Skip-Gram with negative sampling, which approach is best for handling rare words?
medium
A. Use subsampling of frequent words.
B. Increase the number of negative samples.
C. Increase the context window size.
D. Use a higher learning rate for rare words.

Solution

  1. Step 1: Understand the problem with rare words

    Rare words can be overshadowed by frequent words, leading to poor embeddings.

  2. Step 2: Evaluate solutions

    Subsampling frequent words reduces their dominance, allowing rare words to have more influence during training.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Subsampling frequent words helps rare words [OK]
Hint: Subsampling frequent words helps rare words [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard