Bird
Raised Fist0

The Diverging Descent

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Diverging Descent
The issue started after a recent code refactor aimed at optimizing the gradient descent implementation.

A linear regression model using gradient descent to predict house prices based on features like size, location, and age. The model is trained on a dataset of 10,000 samples.

Symptom
The model's loss increases with each iteration of gradient descent, leading to diverging weights and predictions.
📡
Signals
metric
Training loss observed during gradient descent iterations.
Loss increases from 0.5 to 10.0 over 100 iterations.
log
Logs from the gradient descent loop showing weight updates.
Weights become increasingly large and negative.
code
Recent changes in the gradient descent implementation.
Refactored code to use a different learning rate schedule.
distribution
Distribution of feature values in the training dataset.
Features are not normalized, with ranges varying from 1 to 1000.
🔬
Hypotheses
The learning rate is too high, causing divergence.
Supporting Signals
metriclog
Verify: Check the learning rate value and its effect on weight updates.
print(f'Learning rate: {learning_rate}')
Features are not normalized, leading to unstable updates.
Supporting Signals
distribution
Verify: Inspect feature scaling and normalize if necessary.
print(df.describe())
🎯
Root Cause
The learning rate was set too high after the refactor, causing the gradient descent to diverge.
Why It Happened

The refactor inadvertently increased the learning rate without proper testing.

🔧
Fix Strategy
ImmediateRevert to the previous stable learning rate.
Short-termImplement a learning rate decay schedule to stabilize updates.
Long-termAdd automated tests for learning rate stability and convergence.
🐍
Python Fix
Python
learning_rate = 0.01  # Revert to a stable learning rate
for i in range(num_iterations):
    predictions = np.dot(X, weights)
    errors = predictions - y
    gradient = np.dot(X.T, errors) / len(y)
    weights -= learning_rate * gradient
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to too many features.
Why candidates think this: Increasing loss might suggest overfitting.
Why wrong: Overfitting typically results in low training loss but high validation loss, not diverging training loss.
The dataset is too small, causing instability.
Why candidates think this: Small datasets can lead to noisy gradients.
Why wrong: The dataset size is sufficient, and the issue is with the learning rate.
📊
Monitoring to Add
  • Monitor training loss for divergence patterns.
  • Log weight updates to detect abnormal changes.
  • Track learning rate values and their impact on convergence.
📋
Process Change

Require peer review for changes to critical hyperparameters like learning rate.

🏗️
Architectural Change

Implement a robust hyperparameter tuning framework to automatically adjust learning rates.

💡
Key Insight
A learning rate that's too high can cause gradient descent to diverge, leading to increasing loss and unstable model weights.

Practice

(1/5)
1. 
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6]])
X_meaned = X - np.mean(X, axis=0)
cov_matrix = np.cov(X_meaned, rowvar=False)
print(cov_matrix)
What does this code output?
easy
A. A 2x3 covariance matrix
B. A 3x3 covariance matrix
C. A 2x2 covariance matrix
D. A 3x2 covariance matrix

Solution

  1. Step 1: Understand the covariance matrix calculation

    The covariance matrix is calculated for the features (columns) of the dataset.

  2. Step 2: Eliminate wrong options

    A 3x3 covariance matrix is incorrect because the covariance matrix is not 3x3 for a 2-feature dataset. A 2x3 covariance matrix and D are incorrect because covariance matrices are square.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Covariance matrix is square, size = features. [OK]
Hint: Covariance matrix is square, size = features. [OK]
Common Mistakes:
2. What is the time complexity of computing the scaled dot-product attention for a single query with n keys?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Identify operations

    For a single query, the main operations are dot product and softmax, both O(n).

  2. Step 2: Eliminate incorrect complexities

    O(n^2) and O(n log n) are incorrect as they imply more complex operations than present.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Single query, n keys -> O(n) [OK]
Hint: Single query, n keys -> O(n) [OK]
Common Mistakes:
3. What does the following Python code snippet output when implementing the Skip-Gram model with negative sampling? 
import numpy as np

vocab_size = 5
embedding_dim = 3

# Initialize random embeddings
embeddings = np.random.rand(vocab_size, embedding_dim)

# Select a word index
word_index = 2

# Get the embedding for the word
word_embedding = embeddings[word_index]

print(word_embedding.shape)
easy
A. (3, 5)
B. (5, 3)
C. (3,)
D. (1, 3)

Solution

  1. Step 1: Understand the code

    The code initializes a random embedding matrix of shape (vocab_size, embedding_dim) and retrieves the embedding for a specific word index.

  2. Step 2: Determine the shape of the retrieved embedding

    The embedding for a single word is a 1D array with length equal to embedding_dim, hence the shape is (3,).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Single word embedding shape is (embedding_dim,) [OK]
Hint: Single word embedding shape is (embedding_dim,) [OK]
Common Mistakes:
4. How can you optimize the computation of a confusion matrix for a large dataset with millions of predictions?
hard
A. Use a more efficient data structure like a linked list.
B. Use a parallel processing library like Dask.
C. Reduce the number of classes to simplify computation.
D. Use a GPU to accelerate matrix operations.

Solution

  1. Step 1: Identify bottleneck

    Large datasets require efficient computation, often limited by CPU processing.

  2. Step 2: Choose optimization strategy

    Parallel processing with Dask can distribute computation across cores.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Parallel processing speeds up large data tasks. [OK]
Hint: Parallel processing speeds up large data tasks. [OK]
Common Mistakes:
5. How can you optimize the memory usage of a NumPy-based backpropagation implementation for a large neural network?
hard
A. Use in-place operations to reduce memory overhead.
B. Use Python lists instead of NumPy arrays.
C. Store all intermediate results for debugging.
D. Increase the batch size to reduce iterations.

Solution

  1. Step 1: Identify memory bottlenecks

    Intermediate results can consume significant memory.

  2. Step 2: Optimize memory usage

    In-place operations reduce memory overhead by modifying data directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    In-place operations save memory [OK]
Hint: In-place operations save memory [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard