Bird
Raised Fist0

The Diverging Descent

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Diverging Descent
Started after the latest code deployment last Monday.

A logistic regression model predicting user churn for a subscription service, trained on a dataset of 100,000 users.

Symptom
Model loss increases with each epoch during training, instead of decreasing.
📡
Signals
metric
Training loss observed in logs during model training.
Training loss starts at 0.693 and increases to 1.5 over 10 epochs.
log
Gradient values logged during training.
Gradients are consistently large, e.g., weights update by 0.5 per step.
code
Learning rate parameter in the training script.
Learning rate set to 1.0, unusually high for logistic regression.
distribution
Distribution of predicted probabilities on the training set.
Predicted probabilities are mostly 0.0 or 1.0, indicating saturation.
metric
Validation loss observed in logs during model training.
Validation loss mirrors training loss, increasing similarly.
🔬
Hypotheses
The learning rate is too high, causing the model to diverge.
Supporting Signals
Training loss increases with each epochGradients are consistently largeLearning rate set to 1.0
Verify: Check if reducing the learning rate stabilizes the training loss.
learning_rate = 0.01; # Reduce and retrain to observe loss behavior
The data preprocessing step is incorrect, leading to poor model input.
Supporting Signals
Predicted probabilities are mostly 0.0 or 1.0
Verify: Inspect data preprocessing pipeline for errors.
print(data.head()) # Check for anomalies in input features
🎯
Root Cause
The learning rate was set too high, causing gradient updates to overshoot the optimal point.
Why It Happened

A recent code change inadvertently increased the learning rate parameter.

🔧
Fix Strategy
ImmediateRollback to the previous model version with a stable learning rate.
Short-termAdjust the learning rate to a more appropriate value (e.g., 0.01) and retrain the model.
Long-termImplement a learning rate scheduler to dynamically adjust the learning rate during training.
🐍
Python Fix
Python
learning_rate = 0.01
# Implement a learning rate scheduler
from torch.optim.lr_scheduler import StepLR
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
✗
Common Wrong Diagnoses
▼ Show
The model architecture is incorrect.
Why candidates think this: The symptoms of increasing loss could suggest a fundamental flaw in the model design.
Why wrong: The model architecture was unchanged; only the learning rate was modified.
The dataset is corrupted.
Why candidates think this: Corrupted data could lead to unexpected training behavior.
Why wrong: Data inspection showed no anomalies; the issue was with the learning rate.
📊
Monitoring to Add
  • Monitor training and validation loss trends for divergence.
  • Log and alert on unusually large gradient updates.
  • Track changes to hyperparameters like learning rate in version control.
📋
Process Change

Introduce a code review checklist that includes verification of hyperparameter changes.

🏗️
Architectural Change

Adopt a configuration management system to track and validate hyperparameter settings across deployments.

💡
Key Insight
A high learning rate can cause gradient descent to diverge, leading to increasing loss during training.

Practice

(1/5)
1. What does the following Python code output when applied to an image array img? 
import numpy as np

def flip_image(img):
    return np.flip(img, axis=1)

img = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
flipped_img = flip_image(img)
print(flipped_img)
easy
A. [[3, 2, 1], [6, 5, 4], [9, 8, 7]]
B. [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
C. [[7, 8, 9], [4, 5, 6], [1, 2, 3]]
D. [[9, 8, 7], [6, 5, 4], [3, 2, 1]]

Solution

  1. Step 1: Understand the flip operation

    The np.flip function flips the array along the specified axis. Here, axis=1 means flipping horizontally.

  2. Step 2: Apply the flip to the array

    Flipping horizontally changes the order of elements in each row.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Horizontal flip reverses row elements. [OK]
Hint: Horizontal flip reverses row elements. [OK]
Common Mistakes:
2. What does the following Python code output?
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

x = np.array([2.0, 1.0, 0.1])
print(softmax(x))
easy
A. [0.659, 0.242, 0.099]
B. [0.5, 0.3, 0.2]
C. [0.665, 0.244, 0.091]
D. [0.7, 0.2, 0.1]

Solution

  1. Step 1: Calculate softmax

    Compute the exponentials of the input array after subtracting the max value for numerical stability.

  2. Step 2: Normalize

    Divide each exponential by the sum of all exponentials to get probabilities.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Softmax outputs sum to 1 [OK]
Hint: Softmax outputs sum to 1 [OK]
Common Mistakes:
3. In a custom dropout layer implementation, how should you modify the forward pass for inference mode to ensure correct scaling?
medium
A. Divide inputs by (1 - dropout_rate).
B. Multiply inputs by (1 - dropout_rate).
C. Apply the same dropout mask as in training.
D. Use a different dropout rate for inference.

Solution

  1. Step 1: Understand Inference Scaling

    During inference, inputs should be scaled by dividing by (1 - dropout_rate) to maintain expected values.

  2. Step 2: Eliminate Wrong Options

    Apply the same dropout mask as in training. incorrectly applies dropout during inference. Multiply inputs by (1 - dropout_rate). scales incorrectly. Use a different dropout rate for inference. suggests changing dropout rate, which is not standard practice.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Scale by dividing during inference. [OK]
Hint: Scale by dividing during inference. [OK]
Common Mistakes:
4. You implemented Robust Scaling but your model's performance dropped. What is the most likely cause?
medium
A. Robust Scaling is not suitable for normally distributed data.
B. Robust Scaling requires data to be centered around zero.
C. Robust Scaling scales data to a range of [0, 1], which is incorrect.
D. Robust Scaling removes outliers, which are important for your model.

Solution

  1. Step 1: Understand Robust Scaling

    Robust Scaling uses the median and IQR, which reduces the influence of outliers.

  2. Step 2: Identify the Issue

    If outliers are important, reducing their influence can harm model performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Robust Scaling reduces outlier impact. [OK]
Hint: Robust Scaling reduces outlier impact. [OK]
Common Mistakes:
5. 
def calculate_f1(confusion_matrix):
    tp = confusion_matrix[0, 0]
    fp = confusion_matrix[1, 0]
    fn = confusion_matrix[0, 1]
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1_score = 2 * (precision * recall) / (precision + recall)
    return f1_score

confusion_matrix = np.array([[50, 10], [5, 35]])
print(calculate_f1(confusion_matrix))
medium
A. 0.9091
B. 0.8696
C. 0.8333
D. Throws a division by zero error

Solution

  1. Step 1: Calculate precision and recall

    Precision = 0.9091, Recall = 0.8333.

  2. Step 2: Calculate F1-score

    F1 = 2 * (0.9091 * 0.8333) / (0.9091 + 0.8333) = 0.8696.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    F1 combines precision and recall. [OK]
Hint: F1 combines precision and recall. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard