Bird
Raised Fist0

The Diverging Gradient Mystery

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Diverging Gradient Mystery
Started occurring after the last model architecture update two weeks ago.

A transformer-based NLP model for sentiment analysis deployed to process 500k tweets per day.

Symptom
Model training loss diverges after the first few epochs, leading to NaN values in gradients.
📡
Signals
metric
Training loss starts increasing rapidly after epoch 3.
Loss increases from 0.5 to NaN by epoch 5.
log
Gradient values logged during training.
Gradients contain NaN values starting from epoch 4.
code
Review of the positional encoding implementation.
Sinusoidal encoding uses division by zero in certain cases.
distribution
Distribution of input sequence lengths.
Sequence lengths vary widely, with some sequences being very short.
metric
Validation accuracy remains stable initially.
Validation accuracy is stable at 0.85 until epoch 3.
🔬
Hypotheses
The sinusoidal positional encoding is causing numerical instability.
Supporting Signals
codelog
Verify: Check for division by zero or very small numbers in the encoding computation.
import numpy as np
positions = np.arange(0, 100)
divisor = np.power(10000, (2 * (positions // 2)) / 100)
# Check for zero or near-zero values in divisor
print('Min divisor:', np.min(divisor))
The model's learning rate is too high, causing divergence.
Supporting Signals
metriclog
Verify: Check the learning rate schedule and compare with previous stable runs.
learning_rate = 0.01
print('Learning rate:', learning_rate)
🎯
Root Cause
The sinusoidal positional encoding implementation has a division by zero error for certain input lengths.
Why It Happened

An update to the positional encoding logic introduced a bug where division by zero occurs for short sequences.

🔧
Fix Strategy
ImmediateRollback to the previous stable model version.
Short-termCorrect the division logic in the sinusoidal positional encoding implementation.
Long-termImplement unit tests for positional encoding to catch numerical instability.
🐍
Python Fix
Python
import numpy as np

def sinusoidal_encoding(position, d_model):
    angle_rates = 1 / np.power(10000, (2 * (np.arange(d_model) // 2)) / np.float32(d_model))
    angle_rads = position[:, np.newaxis] * angle_rates[np.newaxis, :]
    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
    return angle_rads

# Ensure no division by zero
positions = np.arange(0, 100)
d_model = 512
encoding = sinusoidal_encoding(positions, d_model)
print('Encoding shape:', encoding.shape)
✗
Common Wrong Diagnoses
▼ Show
The model architecture is too complex, leading to overfitting.
Why candidates think this: The divergence in training loss could suggest overfitting.
Why wrong: Overfitting would not cause NaN gradients; the issue is with numerical stability.
The input data distribution has changed significantly.
Why candidates think this: A change in input data could affect model performance.
Why wrong: The issue is specific to the encoding logic, not the input data.
📊
Monitoring to Add
  • Monitor gradient values for NaN or Inf during training.
  • Add alerts for sudden spikes in training loss.
  • Track distribution of input sequence lengths and flag anomalies.
  • Log and review positional encoding outputs for numerical stability.
📋
Process Change

Introduce code reviews specifically for numerical stability in mathematical computations.

🏗️
Architectural Change

Adopt a modular testing framework for all encoding functions to ensure correctness.

💡
Key Insight
Numerical stability in encoding functions is crucial to prevent gradient divergence in deep learning models.

Practice

(1/5)
1. What is the time complexity of computing a confusion matrix for a binary classification problem with n predictions?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Analyze the operation

    Computing a confusion matrix involves iterating over predictions once.

  2. Step 2: Determine complexity

    Each prediction is checked once, leading to O(n) complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Confusion matrix computation is linear in predictions. [OK]
Hint: Confusion matrix computation is linear in predictions. [OK]
Common Mistakes:
2. What is the time complexity of calculating precision, recall, and F1-score from a confusion matrix?
easy
A. O(1)
B. O(n)
C. O(n^2)
D. O(log n)

Solution

  1. Step 1: Analyze the operations

    Calculating precision, recall, and F1-score involves basic arithmetic operations on a fixed-size matrix.

  2. Step 2: Determine complexity

    These operations are constant time, O(1), as they do not depend on the size of the input data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Fixed-size matrix operations are O(1). [OK]
Hint: Fixed-size matrix operations are O(1). [OK]
Common Mistakes:
3. What is the time complexity of computing the softmax function for an input array of size n?
easy
A. O(n^2)
B. O(n log n)
C. O(1)
D. O(n)

Solution

  1. Step 1: Analyze operations

    Softmax involves computing exponentials and a sum, both linear in n.

  2. Step 2: Determine complexity

    Each operation is O(n), so overall complexity is O(n).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Softmax is linear in input size [OK]
Hint: Softmax is linear in input size [OK]
Common Mistakes:
4. 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 1, 1, 0, 1])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, stratify=y)
print(y_train)
easy
A. [0, 0, 1]
B. [0, 1, 0]
C. [1, 0, 1]
D. [0, 1, 1]

Solution

  1. Step 1: Understand stratification

    Stratification ensures that the class distribution in the train and test sets is similar to the original dataset.

  2. Step 2: Calculate class distribution

    Original y has 3 zeros and 3 ones. With stratification, y_train should have 1 zero and 2 ones.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stratify maintains class distribution. [OK]
Hint: Stratify maintains class distribution. [OK]
Common Mistakes:
5. How can you optimize the backward pass of batch normalization to improve computational efficiency?
hard
A. Fuse batch normalization with preceding linear layers.
B. Pre-compute gradients for all layers.
C. Use a smaller batch size to reduce computation.
D. Use a higher learning rate to speed up convergence.

Solution

  1. Step 1: Identify optimization techniques

    Fusing operations can reduce computational overhead.

  2. Step 2: Evaluate each option

    Fusing batch normalization with linear layers reduces redundant computations.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Fuse layers to optimize batch norm [OK]
Hint: Fuse layers to optimize batch norm [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard