Bird
Raised Fist0

Linear Regression - From Least Squares to Gradient Descent

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Gradient Descent Anomaly
Started after the last model update two days ago.

A linear regression model predicting house prices based on features like size, location, and age, serving a real estate platform with 500K daily users.

Symptom
Model accuracy is acceptable, but recall is consistently 0 across multiple test sets.
📡
Signals
metric
Recall on test set
Recall is 0.0 across all test sets
metric
Precision on test set
Precision is 0.85, indicating some correct predictions
feature_importance
Feature importance ranking
Feature 'house_size' importance dropped from 0.25 to 0.05 after update
log
Training log for gradient descent
Gradient values are extremely large, indicating potential divergence
distribution
Distribution of predicted values
Predicted values are all clustered around a single point
🔬
Hypotheses
Gradient descent is diverging due to a high learning rate
Supporting Signals
Gradient values are extremely large
Verify: Check the learning rate used in the gradient descent algorithm
print(model.learning_rate)
Feature scaling issue causing poor convergence
Supporting Signals
Feature 'house_size' importance dropped significantly
Verify: Check if features are properly scaled before training
print(df.describe())
🎯
Root Cause
Gradient descent diverged due to an excessively high learning rate set during the last update.
Why It Happened

A configuration error in the deployment script set the learning rate to a value 10x higher than intended.

🔧
Fix Strategy
ImmediateRollback to the previous model version with a stable learning rate.
Short-termCorrect the learning rate in the configuration and retrain the model.
Long-termImplement a validation step in the deployment pipeline to check learning rate settings.
🐍
Python Fix
Python
model.learning_rate = 0.01
model.fit(X_train, y_train)
✗
Common Wrong Diagnoses
▼ Show
Feature engineering error causing recall drop
Why candidates think this: Feature importance changes suggest a feature issue.
Why wrong: The feature importance drop is a result of poor model convergence, not the cause.
Data leakage causing recall to be 0
Why candidates think this: Recall of 0 could suggest a data issue.
Why wrong: Data leakage would likely affect both precision and recall, not just recall.
📊
Monitoring to Add
  • Monitor gradient values during training to detect divergence early.
  • Set up alerts for unexpected changes in feature importance.
  • Track distribution of predicted values to catch clustering issues.
📋
Process Change

Introduce a peer review step for configuration changes before deployment.

🏗️
Architectural Change

Add automated tests to validate hyperparameter settings before training begins.

💡
Key Insight
A high learning rate can cause gradient descent to diverge, leading to unexpected model behavior despite seemingly correct configurations.

Practice

(1/5)
1. In the context of data labeling strategies, which method is best suited for generating large amounts of labeled data quickly but with potentially lower accuracy?
easy
A. Weak Supervision
B. Programmatic Labeling
C. Human Annotation
D. Active Learning

Solution

  1. Step 1: Understand the methods.

    Human annotation is accurate but slow. Weak supervision uses heuristics or external data sources to generate labels quickly but may be less accurate.

  2. Step 2: Eliminate wrong options.

    Programmatic labeling is similar to weak supervision but often requires more setup. Active learning involves human input and is not as fast.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Weak supervision = fast, less accurate [OK]
Hint: Weak supervision = fast, less accurate [OK]
Common Mistakes:
2. Given the following Python code snippet using weak supervision, what will be the output?
import numpy as np
labels = np.array(['cat', 'dog', 'cat', 'dog'])
heuristics = np.array([1, 0, 1, 0])
weak_labels = np.where(heuristics == 1, 'cat', 'dog')
print(weak_labels)
easy
A. ['cat', 'dog', 'cat', 'dog']
B. ['cat', 'cat', 'cat', 'cat']
C. ['cat', 'dog', 'cat', 'dog']
D. ['cat', 'dog', 'cat', 'dog']

Solution

  1. Step 1: Analyze the code.

    The code uses np.where to assign 'cat' where heuristics is 1 and 'dog' where it is 0.

  2. Step 2: Determine the output.

    Since heuristics is [1, 0, 1, 0], the output will be ['cat', 'dog', 'cat', 'dog'].

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    np.where logic: match heuristics [OK]
Hint: np.where logic: match heuristics [OK]
Common Mistakes:
3. 
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
X = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
y = np.array([1, 0, 1])
selector = SelectKBest(score_func=f_classif, k=2)
X_new = selector.fit_transform(X, y)
print(X_new)
What does the above code output?
easy
A. array([[2, 3], [5, 6], [8, 9]])
B. array([[1, 3], [4, 6], [7, 9]])
C. array([[1, 2], [4, 5], [7, 8]])
D. array([[3, 2], [6, 5], [9, 8]])

Solution

  1. Step 1: Understand SelectKBest.

    SelectKBest selects the top k features based on the highest scores from the scoring function.

  2. Step 2: Determine the output.

    f_classif scores features based on ANOVA F-value. Here, features 2 and 3 have the highest scores.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Top k features by score. [OK]
Hint: SelectKBest picks top k features by score. [OK]
Common Mistakes:
4. In the context of Naive Bayes, which of the following best describes the assumption of conditional independence?
easy
A. Features are independent of the class label.
B. Features are independent given the class label.
C. Features are dependent on each other given the class label.
D. Features are independent in all scenarios.

Solution

  1. Step 1: Understand Conditional Independence

    Naive Bayes assumes that features are independent given the class label.

  2. Step 2: Eliminate Wrong Options

    Features are independent of the class label. suggests features are independent of the class label, which is incorrect. Features are dependent on each other given the class label. suggests dependency, which contradicts the assumption. Features are independent in all scenarios. suggests independence in all scenarios, which is overly broad.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Naive Bayes assumes independence given class. [OK]
Hint: Naive Bayes assumes independence given class. [OK]
Common Mistakes:
5. When debugging a model with high variance, which approach is more suitable?
medium
A. Increase model complexity
B. Add more training data
C. Reduce regularization
D. Use a smaller model

Solution

  1. Step 1: Identify high variance symptoms

    High variance indicates overfitting, where the model performs well on training data but poorly on unseen data.

  2. Step 2: Choose the right approach

    Adding more training data helps the model generalize better, reducing overfitting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High variance = more data [OK]
Hint: High variance = more data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard