Bird
Raised Fist0

The Divergent Gradient Descent

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Divergent Gradient Descent
Started occurring after the latest deployment last week.

A neural network model for image classification, deployed to classify 1M images per day.

Symptom
Model training loss diverges after a few epochs, causing training to halt unexpectedly.
📡
Signals
metric
Training loss increases exponentially after the third epoch.
Training loss jumps from 0.5 to 10.0 between epochs 3 and 4.
log
Gradient values logged during training.
Gradients contain NaN values after the third epoch.
code
Review of the softmax and cross-entropy loss implementation.
Softmax implementation lacks numerical stability measures.
distribution
Distribution of predicted probabilities.
Predicted probabilities are extremely skewed, with most values near 0 or 1.
🔬
Hypotheses
Numerical instability in the softmax function is causing NaN gradients.
Supporting Signals
Gradients contain NaN values after the third epoch.Softmax implementation lacks numerical stability measures.
Verify: Check if the softmax function implementation uses a numerically stable approach.
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

# Test with large input values
x = np.array([1000, 1000, 1000])
print(softmax(x))
Incorrect gradient computation due to wrong loss function implementation.
Supporting Signals
Training loss increases exponentially after the third epoch.Predicted probabilities are extremely skewed.
Verify: Review the cross-entropy loss function for correct gradient computation.
import numpy as np

def cross_entropy_loss(y_true, y_pred):
    # Add a small epsilon to prevent log(0)
    epsilon = 1e-15
    y_pred = np.clip(y_pred, epsilon, 1. - epsilon)
    return -np.sum(y_true * np.log(y_pred))

# Test with valid probabilities
y_true = np.array([1, 0, 0])
y_pred = np.array([0.7, 0.2, 0.1])
print(cross_entropy_loss(y_true, y_pred))
🎯
Root Cause
Numerical instability in the softmax function leading to NaN gradients.
Why It Happened

The softmax function was implemented without considering numerical stability, causing overflow with large input values.

🔧
Fix Strategy
ImmediateRollback to the previous stable model version.
Short-termImplement a numerically stable softmax function.
Long-termIntegrate numerical stability checks in all critical mathematical operations.
🐍
Python Fix
Python
import numpy as np

def stable_softmax(x):
    shiftx = x - np.max(x)
    exps = np.exp(shiftx)
    return exps / np.sum(exps)

# Use this stable version in the model
x = np.array([1000, 1000, 1000])
print(stable_softmax(x))
✗
Common Wrong Diagnoses
▼ Show
The learning rate is too high, causing divergence.
Why candidates think this: High learning rates can cause training instability.
Why wrong: The issue persisted even with reduced learning rates, indicating a different root cause.
The model architecture is too complex, leading to overfitting.
Why candidates think this: Complex models can overfit and exhibit unstable training.
Why wrong: The divergence occurred too early in training, unrelated to model complexity.
📊
Monitoring to Add
  • Monitor gradient values for NaNs during training.
  • Track the distribution of predicted probabilities for extreme skewness.
  • Log the maximum input values to the softmax function.
📋
Process Change

Require code reviews for numerical stability in mathematical operations.

🏗️
Architectural Change

Adopt a library or framework that inherently handles numerical stability in operations.

💡
Key Insight
Numerical stability in softmax is crucial to prevent gradient divergence in neural networks.

Practice

(1/5)
1. In a scenario where training time is critical, which approach is better for implementing early stopping?
medium
A. Use a small patience value to stop early.
B. Use a large patience value to ensure stability.
C. Monitor training loss instead of validation loss.
D. Disable early stopping to avoid interruptions.

Solution

  1. Step 1: Consider training time constraints

    In time-critical scenarios, stopping early can save resources.

  2. Step 2: Choose the appropriate patience value

    A small patience value allows for quicker termination if no improvement is seen, aligning with the need to save time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Small patience saves time in critical scenarios. [OK]
Hint: Small patience saves time in critical scenarios. [OK]
Common Mistakes:
2. 
import numpy as np

# Backpropagation for a single neuron
inputs = np.array([1.0, 2.0, 3.0])
weights = np.array([0.2, 0.8, -0.5])
bias = 2.0
learning_rate = 0.01

d_output = np.array([1.0])  # Gradient of loss w.r.t output

# Incorrect gradient calculation
weights_gradient = d_output * inputs
weights -= learning_rate * weights_gradient
print(weights)
medium
A. Weights are updated correctly.
B. Weights are updated incorrectly due to incorrect learning rate.
C. Weights are not updated; learning rate is too small.
D. Weights are updated incorrectly due to missing chain rule.

Solution

  1. Step 1: Identify error in gradient calculation

    The code incorrectly applies the gradient directly without considering the chain rule.

  2. Step 2: Correct approach

    Should multiply by the derivative of the activation function.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Always apply chain rule in backpropagation [OK]
Hint: Always apply chain rule in backpropagation [OK]
Common Mistakes:
3. In a PCA implementation, you need to project data onto the first two principal components. Which approach is correct?
medium
A. Multiply the data matrix by the first two eigenvectors.
B. Multiply the data matrix by the first two eigenvalues.
C. Multiply the covariance matrix by the first two eigenvectors.
D. Multiply the covariance matrix by the first two eigenvalues.

Solution

  1. Step 1: Understand projection in PCA

    Projection involves multiplying the data matrix by the eigenvectors corresponding to the desired components.

  2. Step 2: Eliminate wrong options

    Multiply the data matrix by the first two eigenvalues. is incorrect because eigenvalues are not used for projection. Multiply the covariance matrix by the first two eigenvectors. and D incorrectly use the covariance matrix instead of the data matrix.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Project data using eigenvectors, not eigenvalues. [OK]
Hint: Project data using eigenvectors, not eigenvalues. [OK]
Common Mistakes:
4. You implemented a Skip-Gram model with negative sampling, but the loss is not decreasing. What is the most likely cause?
medium
A. The embeddings are not initialized randomly.
B. The context window size is too small.
C. The negative samples are not truly random.
D. The learning rate is too high.

Solution

  1. Step 1: Analyze the symptoms

    A non-decreasing loss often indicates issues with learning rate or gradient updates.

  2. Step 2: Evaluate each option

    High learning rates can cause oscillations, preventing loss reduction. Random initialization and sample randomness are less likely to cause this specific issue.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    High learning rate can cause loss oscillation [OK]
Hint: High learning rate can cause loss oscillation [OK]
Common Mistakes:
5. How can you optimize the memory usage of the Adam optimizer in a large-scale deep learning model?
hard
A. Reduce the batch size to decrease memory footprint.
B. Use half-precision floating points for parameter storage.
C. Share momentum and velocity arrays across layers.
D. Store only the last two gradients instead of all past gradients.

Solution

  1. Step 1: Identify memory-intensive components

    Adam requires storing additional momentum and velocity arrays, which can be memory-intensive.

  2. Step 2: Optimize storage precision

    Using half-precision floating points reduces memory usage significantly while maintaining sufficient precision for most deep learning tasks.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Use half-precision to save memory in Adam [OK]
Hint: Use half-precision to save memory in Adam [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard