Bird
Raised Fist0

The Diverging Gradients Mystery

CODINGadvancedInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Diverging Gradients Mystery
This issue started occurring after a recent update to the training script last week.

A Word2Vec model using Skip-Gram with Negative Sampling for a large-scale text corpus processing pipeline.

Symptom
The model's loss starts decreasing initially but then begins to diverge, leading to NaN values in the gradients.
📡
Signals
metric
Training loss observed during model training
Loss decreases initially but then increases and eventually becomes NaN.
log
Gradient values logged during training
Gradients become NaN after a few epochs.
code
Recent changes in the training script
Added a new initialization method for word vectors.
distribution
Distribution of word vector norms
Some word vectors have extremely large norms compared to others.
🔬
Hypotheses
The new initialization method is causing instability in training.
Supporting Signals
Recent changes in the training scriptDistribution of word vector norms
Verify: Check if reverting to the old initialization stabilizes training.
max_norm = np.linalg.norm(word_vectors, axis=1).max(); print('Max norm:', max_norm)
Learning rate is too high, causing divergence.
Supporting Signals
Training loss observed during model trainingGradient values logged during training
Verify: Reduce the learning rate and observe if the loss stabilizes.
learning_rate = 0.01; # Try reducing this value and observe changes in loss
🎯
Root Cause
The new initialization method for word vectors led to large initial values, causing gradient explosion.
Why It Happened

The initialization method was not suitable for the scale of the word vectors, leading to instability.

🔧
Fix Strategy
ImmediateRevert to the previous initialization method for word vectors.
Short-termImplement a more stable initialization method, such as Xavier or He initialization.
Long-termDevelop a robust initialization strategy that adapts to the scale of the data.
🐍
Python Fix
Python
word_vectors = np.random.randn(vocab_size, vector_size) * 0.01  # Revert to small random initialization
✗
Common Wrong Diagnoses
▼ Show
The learning rate is too high.
Why candidates think this: Diverging loss and NaN gradients are common symptoms of a high learning rate.
Why wrong: Reducing the learning rate did not resolve the issue; the problem was with initialization.
The dataset contains corrupted data.
Why candidates think this: Corrupted data can lead to unexpected training behavior.
Why wrong: The issue persisted even with a clean dataset; it was related to initialization.
📊
Monitoring to Add
  • Monitor the distribution of word vector norms during training.
  • Add checks for NaN values in gradients after each epoch.
  • Track changes in training loss patterns to detect divergence early.
📋
Process Change

Require thorough testing of new initialization methods on a smaller dataset before full deployment.

🏗️
Architectural Change

Incorporate adaptive learning rate strategies that adjust based on gradient norms.

💡
Key Insight
Initialization methods can drastically affect training stability, especially in large-scale models.

Practice

(1/5)
1. 
import numpy as np

# Forward pass for a single neuron
inputs = np.array([1.0, 2.0, 3.0])
weights = np.array([0.2, 0.8, -0.5])
bias = 2.0

output = np.dot(inputs, weights) + bias
print(output)
easy
A. 2.8
B. 2.0
C. 3.0
D. 2.3

Solution

  1. Step 1: Calculate dot product

    np.dot(inputs, weights) = 1.0*0.2 + 2.0*0.8 + 3.0*(-0.5) = 0.2 + 1.6 - 1.5 = 0.3

  2. Step 2: Add bias

    0.3 + 2.0 = 2.3

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dot product then add bias [OK]
Hint: Dot product then add bias [OK]
Common Mistakes:
2. What is the time complexity of finding the nearest neighbors using a brute force approach in a KNN classifier?
easy
A. O(n log n)
B. O(n^2)
C. O(n * d)
D. O(n)

Solution

  1. Step 1: Understand brute force KNN

    Brute force KNN involves calculating the distance from the query point to all training points.

  2. Step 2: Analyze complexity

    For each of the n training points, we compute the distance in d dimensions, leading to O(n * d).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Brute force KNN: O(n * d) [OK]
Hint: Brute force KNN: O(n * d) [OK]
Common Mistakes:
3. You implemented stratified K-Fold cross-validation but notice class imbalance in some folds. What is the most likely cause?
medium
A. Incorrect stratification parameter
B. Data shuffling before splitting
C. Incorrect number of splits
D. Too many folds for the dataset size

Solution

  1. Step 1: Identify stratification issue

    Stratification ensures class distribution is preserved in each fold.

  2. Step 2: Eliminate wrong options

    Incorrect number of splits and D relate to fold count, not class distribution. Data shuffling before splitting affects randomness, not stratification.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Check stratification parameter for class balance. [OK]
Hint: Check stratification parameter for class balance. [OK]
Common Mistakes:
4. In which scenario is gradient checking most useful?
medium
A. When implementing a new custom layer in a neural network.
B. When training a model with a large dataset.
C. When optimizing hyperparameters.
D. When deploying a model to production.

Solution

  1. Step 1: Identify scenarios where gradient checking is applicable

    Gradient checking is crucial when implementing new components to ensure correctness.

  2. Step 2: Eliminate unrelated scenarios

    Options A, C, and D are not directly related to verifying gradient correctness.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Use gradient check for new components. [OK]
Hint: Use gradient check for new components. [OK]
Common Mistakes:
5. How can you optimize the computation of eigenvectors for a large covariance matrix in PCA?
hard
A. Use the full SVD decomposition.
B. Use the power iteration method.
C. Use the Gram-Schmidt process.
D. Use incremental PCA.

Solution

  1. Step 1: Identify efficient eigenvector computation methods

    The power iteration method is efficient for finding the largest eigenvectors of large matrices.

  2. Step 2: Eliminate wrong options

    Use the full SVD decomposition. is computationally expensive. Use incremental PCA. is for streaming data, not large matrices. Use the Gram-Schmidt process. is for orthogonalization, not eigenvector computation.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Power iteration for large matrix eigenvectors. [OK]
Hint: Power iteration for large matrix eigenvectors. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard