Bird
Raised Fist0

The Misleading Multinomial

CODINGintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Misleading Multinomial
Started immediately after the last model deployment on Monday

Spam detection model using Naive Bayes serving 500K emails/day

Symptom
Model accuracy dropped from 95% to 70% after last update
📡
Signals
metric
Training accuracy remains high at 96%
Training accuracy is stable at 96% despite drop in production
metric
Validation accuracy dropped significantly
Validation accuracy dropped from 94% to 72% after deployment
distribution
Distribution of predicted probabilities
Predicted probabilities are skewed towards one class
log
Model logs showing class prior probabilities
Class prior probabilities are not updating correctly
code
Recent code changes in the probability calculation
Code change in the calculation of class probabilities
🔬
Hypotheses
Class prior probabilities are not being updated correctly
Supporting Signals
Model logs showing class prior probabilitiesCode change in the calculation of class probabilities
Verify: Check the code responsible for updating class priors
print(model.class_priors)  # Check if class priors are being updated correctly
Feature distribution mismatch between training and production
Supporting Signals
Distribution of predicted probabilities
Verify: Compare feature distributions between training and production datasets
train_features.mean(), prod_features.mean()  # Compare means of features
🎯
Root Cause
Class prior probabilities were hardcoded and not updated with new data
Why It Happened

A recent code change inadvertently fixed class priors instead of recalculating them dynamically

🔧
Fix Strategy
ImmediateRollback to the previous model version
Short-termCorrect the code to dynamically update class priors with each batch of data
Long-termImplement a validation step to ensure class priors are recalculated before deployment
🐍
Python Fix
Python
def update_class_priors(data):
    class_counts = np.bincount(data['class'])
    total_count = len(data['class'])
    return class_counts / total_count

model.class_priors = update_class_priors(training_data)
✗
Common Wrong Diagnoses
▼ Show
Model is overfitting to the training data
Why candidates think this: High training accuracy with low validation accuracy suggests overfitting
Why wrong: The issue is not overfitting but incorrect class prior probabilities
Data leakage between training and validation sets
Why candidates think this: Stable training accuracy with dropping validation accuracy
Why wrong: The validation set is affected by the same fixed priors issue
📊
Monitoring to Add
  • Monitor class prior probabilities after each training epoch
  • Set up alerts for significant drops in validation accuracy
  • Track distribution of predicted probabilities over time
📋
Process Change

Introduce a code review checklist item for dynamic parameter updates

🏗️
Architectural Change

Design the model training pipeline to automatically validate class prior updates

💡
Key Insight
Dynamic updates of class priors are crucial for maintaining model accuracy across different datasets.

Practice

(1/5)
1. 
from sklearn.model_selection import KFold
import numpy as np

X = np.array([1, 2, 3, 4, 5])
kf = KFold(n_splits=2)

for train_index, test_index in kf.split(X):
    print("Train:", train_index, "Test:", test_index)
What will be the output of the above code?
easy
A. Train: [0 1 2 3] Test: [4] and Train: [4] Test: [0 1 2 3]
B. Train: [0 1 2] Test: [3 4] and Train: [3 4] Test: [0 1 2]
C. Train: [0 1] Test: [2 3 4] and Train: [2 3 4] Test: [0 1]
D. Train: [0 1 2 3] Test: [4] and Train: [0 1 2 4] Test: [3]

Solution

  1. Step 1: Understand KFold behavior

    KFold splits the data into n_splits parts, here 2, and iterates over them.

  2. Step 2: Eliminate wrong options

    Train: [0 1 2 3] Test: [4] and Train: [4] Test: [0 1 2 3] and D suggest incorrect split sizes. Train: [0 1] Test: [2 3 4] and Train: [2 3 4] Test: [0 1] suggests incorrect indexing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    KFold splits into equal parts. [OK]
Hint: KFold splits data into equal parts. [OK]
Common Mistakes:
2. What is the time complexity of a single update step in Mini-Batch SGD with Momentum for a model with n parameters?
easy
A. O(n)
B. O(n log n)
C. O(n^2)
D. O(1)

Solution

  1. Step 1: Identify operations involved

    Each update step involves computing the gradient and updating the velocity and parameters, all of which are linear operations with respect to the number of parameters.

  2. Step 2: Determine the complexity

    Since each operation is linear, the overall complexity is O(n).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Each parameter update is linear in complexity [OK]
Hint: Each parameter update is linear in complexity [OK]
Common Mistakes:
3. In a recommendation system, why might you choose cosine similarity over Euclidean distance?
medium
A. Cosine similarity is invariant to vector magnitude.
B. Cosine similarity is more interpretable than Euclidean distance.
C. Cosine similarity handles missing data better.
D. Cosine similarity is faster to compute for large vectors.

Solution

  1. Step 1: Compare properties

    Cosine similarity measures angle, not magnitude, making it invariant to vector length.

  2. Step 2: Eliminate incorrect reasons

    Cosine similarity isn't inherently faster (A), doesn't handle missing data better (C), and interpretability is subjective (D).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cosine ignores magnitude, focuses on direction. [OK]
Hint: Cosine ignores magnitude, focuses on direction. [OK]
Common Mistakes:
4. 
import numpy as np

def ndcg_at_k(r, k):
    dcg_max = dcg_at_k(sorted(r, reverse=True), k)
    if not dcg_max:
        return 0.
    return dcg_at_k(r, k) / dcg_max

relevance_scores = [3, 2, 3, 0, 1, 2]
k = 3
print(ndcg_at_k(relevance_scores, k))
The output is unexpectedly low. What is the most likely cause?
medium
A. The k value is too low.
B. The DCG is calculated incorrectly.
C. The relevance scores are not normalized.
D. The ideal DCG is not calculated correctly.

Solution

  1. Step 1: Check Ideal DCG Calculation

    The ideal DCG should be calculated using the sorted relevance scores.

  2. Step 2: Identify the Error

    If the ideal DCG is not calculated correctly, the NDCG will be incorrect.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Check ideal DCG calculation. [OK]
Hint: Check ideal DCG calculation. [OK]
Common Mistakes:
5. How can you optimize the numerical gradient computation for a large neural network?
hard
A. Use a smaller step size to reduce computation time.
B. Parallelize the gradient computation across multiple cores.
C. Increase the batch size during training.
D. Use a higher learning rate to speed up convergence.

Solution

  1. Step 1: Identify optimization strategies for numerical gradient computation

    Parallelizing the computation can significantly reduce time for large networks.

  2. Step 2: Eliminate ineffective strategies

    Options A, C, and D do not directly optimize the gradient computation process.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Parallelize for large networks. [OK]
Hint: Parallelize for large networks. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard