Bird
Raised Fist0

The Divergent Gradients Mystery

CODINGadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Divergent Gradients Mystery
Started happening after the latest deployment last week.

A neural network model for image classification, deployed to classify 1M images per day.

Symptom
The model's training loss diverges after a few epochs, causing the model to perform poorly on the validation set.
📡
Signals
metric
Training loss over epochs
Loss decreases initially but then starts increasing rapidly after epoch 5.
log
Gradient norms logged during training
Gradient norms are extremely large, exceeding 1000 after epoch 5.
distribution
Weight distribution of the first layer
Weights have a mean of 0 but a standard deviation that increases significantly after epoch 5.
code
Recent changes in the backpropagation implementation
A new line of code was added to scale gradients by a factor of 10.
🔬
Hypotheses
Gradient scaling is causing instability in the training process.
Supporting Signals
Gradient norms are extremely large, exceeding 1000 after epoch 5.A new line of code was added to scale gradients by a factor of 10.
Verify: Check the gradient scaling factor and its impact on gradient norms.
import numpy as np
# Check gradient norms
for epoch in range(epochs):
    gradients = compute_gradients()
    scaled_gradients = gradients * 10
    norm = np.linalg.norm(scaled_gradients)
    print('Epoch:', epoch, 'Gradient norm:', norm)
    if norm > 1000:
        print('Warning: Gradient norm exceeds threshold')
Weight initialization might be causing the divergence.
Supporting Signals
Weights have a mean of 0 but a standard deviation that increases significantly after epoch 5.
Verify: Inspect the weight initialization method and its impact on training stability.
import numpy as np
# Check weight initialization
weights = np.random.randn(layer_size, input_size) * 0.01
for epoch in range(epochs):
    # Simulate weight update
    weights += np.random.randn(*weights.shape) * 0.01
    print('Epoch:', epoch, 'Weight std:', np.std(weights))
🎯
Root Cause
The gradient scaling factor was incorrectly set to 10, causing gradients to explode.
Why It Happened

A recent code change introduced an incorrect scaling factor in the backpropagation step.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the model without the gradient scaling change.
Short-termCorrect the gradient scaling factor to 1 in the backpropagation implementation.
Long-termImplement gradient clipping to prevent exploding gradients in future updates.
🐍
Python Fix
Python
def backpropagation(X, y, weights, learning_rate):
    gradients = compute_gradients(X, y, weights)
    # Correct the scaling factor
    scaled_gradients = gradients * 1
    weights -= learning_rate * scaled_gradients
    return weights
✗
Common Wrong Diagnoses
▼ Show
The learning rate is too high, causing divergence.
Why candidates think this: High learning rates can cause training instability.
Why wrong: The learning rate was unchanged, and the issue was due to incorrect gradient scaling.
Weight initialization is incorrect, leading to divergence.
Why candidates think this: Improper initialization can cause training issues.
Why wrong: The initialization method was standard and unchanged; the problem was with gradient scaling.
📊
Monitoring to Add
  • Monitor gradient norms during training to detect potential instability.
  • Track changes in weight distributions across epochs.
  • Set alerts for sudden increases in training loss.
📋
Process Change

Require code reviews for all changes to the training loop, especially those affecting gradient calculations.

🏗️
Architectural Change

Adopt a modular training framework that includes built-in checks for gradient stability.

💡
Key Insight
Incorrect gradient scaling can lead to exploding gradients, causing training instability.

Practice

(1/5)
1. What is the time complexity of computing a confusion matrix for a binary classification problem with n predictions?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Analyze the operation

    Computing a confusion matrix involves iterating over predictions once.

  2. Step 2: Determine complexity

    Each prediction is checked once, leading to O(n) complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Confusion matrix computation is linear in predictions. [OK]
Hint: Confusion matrix computation is linear in predictions. [OK]
Common Mistakes:
2. 
import numpy as np

class DropoutLayer:
    def __init__(self, dropout_rate):
        self.dropout_rate = dropout_rate

    def forward(self, inputs, training=True):
        if training:
            self.mask = np.random.binomial(1, 1 - self.dropout_rate, size=inputs.shape)
            return inputs * self.mask
        else:
            return inputs

inputs = np.array([1.0, 2.0, 3.0])
dropout_layer = DropoutLayer(0.5)
output = dropout_layer.forward(inputs, training=False)
print(output)
easy
A. [0.5, 1.0, 1.5]
B. [1.0, 2.0, 3.0]
C. [0.0, 0.0, 0.0]
D. [2.0, 4.0, 6.0]

Solution

  1. Step 1: Understand Inference Mode

    During inference, dropout is not applied, so the input is returned unchanged.

  2. Step 2: Eliminate Wrong Options

    [0.5, 1.0, 1.5] applies scaling, which is incorrect for inference. [0.0, 0.0, 0.0] assumes all zeros, which is not the case. [2.0, 4.0, 6.0] doubles the input, which is incorrect.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Inference mode returns inputs unchanged. [OK]
Hint: Inference mode returns inputs unchanged. [OK]
Common Mistakes:
3. 
import numpy as np

def dcg_at_k(r, k):
    r = np.asfarray(r)[:k]
    if r.size:
        return np.sum(r / np.log2(np.arange(2, r.size + 2)))
    return 0.0

relevance_scores = [3, 2, 3, 0, 1, 2]
k = 3
print(dcg_at_k(relevance_scores, k))
What does this code output?
easy
A. 5.2618595071429155
B. 4.2618595071429155
C. 3.2618595071429155
D. 6.2618595071429155

Solution

  1. Step 1: Understand DCG Calculation

    The DCG is calculated as the sum of relevance scores divided by log base 2 of their positions.

  2. Step 2: Calculate for k=3

    For k=3, the scores are [3, 2, 3]. The calculation is 3/1 + 2/log2(3) + 3/log2(4).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DCG uses log2 positions. [OK]
Hint: DCG uses log2 positions. [OK]
Common Mistakes:
4. Your mAP implementation is returning unusually high values. What is the most likely cause?
medium
A. Using precision instead of recall in calculations
B. Incorrect IoU threshold for positive matches
C. Not normalizing the precision values
D. Including false negatives in the precision calculation

Solution

  1. Step 1: Identify the Symptom

    High mAP values suggest too many matches are considered correct.

  2. Step 2: Analyze Each Option

    Incorrect IoU threshold for positive matches: A low IoU threshold can cause false positives to be counted as true positives, inflating mAP. Using precision instead of recall in calculations: Would affect recall, not mAP directly. Not normalizing the precision values: Would affect the scale, not the value. Including false negatives in the precision calculation: Would lower precision, not increase it.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Check IoU threshold for false positives [OK]
Hint: Check IoU threshold for false positives [OK]
Common Mistakes:
5. How can you optimize the following non-max suppression code for large datasets? 
def non_max_suppression(boxes, scores, iou_threshold):
    indices = np.argsort(scores)[::-1]
    keep = []
    while len(indices) > 0:
        current = indices[0]
        keep.append(current)
        indices = indices[1:]
        indices = [i for i in indices if iou(boxes[current], boxes[i]) < iou_threshold]
    return keep
hard
A. Parallelize the IoU calculations.
B. Use a more efficient sorting algorithm.
C. Reduce the IoU threshold.
D. Use a different data structure for indices.

Solution

  1. Step 1: Identify bottleneck

    IoU calculations are the most computationally expensive part.

  2. Step 2: Optimize bottleneck

    Parallelizing IoU calculations can significantly reduce runtime for large datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Parallelize expensive operations [OK]
Hint: Parallelize expensive operations [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard