Bird
Raised Fist0

Ranking Metrics - NDCG, MAP, MRR - How Search and Rec Systems Measure Quality

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Mysterious Recall Drop
The issue started after the latest model update last Friday.

A recommendation system for an e-commerce platform serving personalized product lists to 500k users daily.

Symptom
The system's recall metric dropped to 0, while precision remained stable.
📡
Signals
metric
Recall metric for the recommendation system
Recall dropped from 0.65 to 0.00 after the update
metric
Precision metric for the recommendation system
Precision remained stable at 0.72
log
Deployment logs showing changes in model configuration
New ranking model deployed with updated hyperparameters
feature_importance
Feature importance analysis post-deployment
Feature 'user_purchase_history' importance dropped from 0.25 to 0.01
distribution
Distribution of predicted scores
Scores are heavily skewed towards lower values post-update
🔬
Hypotheses
The model's threshold for positive predictions is set too high.
Supporting Signals
Recall dropped to 0Scores skewed towards lower values
Verify: Check the threshold setting for positive predictions.
import numpy as np
threshold = 0.9
predictions = (scores > threshold).astype(int)
recall = np.sum((predictions == 1) & (true_labels == 1)) / np.sum(true_labels == 1)
Critical features were misconfigured or dropped during the update.
Supporting Signals
Feature 'user_purchase_history' importance dropped
Verify: Review feature engineering pipeline and model input features.
important_features = ['user_purchase_history']
missing_features = [f for f in important_features if f not in model_input_features]
print('Missing features:', missing_features)
🎯
Root Cause
The model's threshold for classifying positive predictions was inadvertently set too high during the update.
Why It Happened

A configuration error in the deployment script set the threshold to an incorrect value.

🔧
Fix Strategy
ImmediateRollback to the previous model version with correct threshold settings.
Short-termCorrect the threshold setting in the deployment script and redeploy.
Long-termImplement a configuration validation step in the deployment pipeline to catch such errors.
🐍
Python Fix
Python
def deploy_model(model, threshold=0.5):
    # Ensure threshold is set correctly
    assert 0 <= threshold <= 1, 'Threshold must be between 0 and 1'
    model.threshold = threshold
    # Deploy the model
    deploy_to_production(model)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to recent data changes.
Why candidates think this: Overfitting could explain a drop in recall if the model is too specific.
Why wrong: Precision remained stable, indicating the model's specificity hasn't changed.
A bug in the feature extraction pipeline caused missing features.
Why candidates think this: Feature importance drop suggests missing features.
Why wrong: The feature extraction pipeline was verified and unchanged.
📊
Monitoring to Add
  • Monitor recall and precision metrics post-deployment for significant deviations.
  • Set up alerts for drastic changes in feature importance.
  • Track distribution of predicted scores to detect skewness.
📋
Process Change

Introduce a checklist for deployment that includes verification of all configuration parameters.

🏗️
Architectural Change

Adopt a feature flag system to safely test configuration changes in production.

💡
Key Insight
A misconfigured threshold can silently cripple recall without affecting precision, highlighting the need for robust deployment checks.

Practice

(1/5)
1. What will be the output shape of the following code snippet using K-Fold cross-validation with 5 splits on a dataset of 100 samples?
from sklearn.model_selection import KFold
kf = KFold(n_splits=5)
for train_index, test_index in kf.split(range(100)):
    print(len(test_index))
easy
A. 20, 20, 20, 20, 20
B. 25, 25, 25, 25
C. 20, 20, 20, 20, 20, 20
D. 20, 20, 20, 20

Solution

  1. Step 1: Calculate the number of samples per fold.

    100 samples divided by 5 splits results in 20 samples per fold.

  2. Step 2: Eliminate wrong options.

    25 samples per fold (B) is incorrect, 6 outputs (C) is incorrect, 4 outputs (D) is incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    100 samples, 5 splits = 20 per fold. [OK]
Hint: 100 samples, 5 splits = 20 per fold. [OK]
Common Mistakes:
2. In the context of recurrent neural networks, which mechanism is specifically designed to address the vanishing gradient problem?
easy
A. Long Short-Term Memory (LSTM)
B. Batch Normalization
C. Weight Decay
D. Dropout

Solution

  1. Step 1: Identify the mechanism for vanishing gradient.

    LSTMs are specifically designed to handle vanishing gradients by using gating mechanisms.

  2. Step 2: Eliminate wrong options.

    Dropout (A) is for regularization, Batch Normalization (B) stabilizes learning, and Weight Decay (D) is for preventing overfitting.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    LSTM handles vanishing gradients with gates. [check]
Hint: LSTM handles vanishing gradients with gates. [check]
Common Mistakes:
3. In which scenario is supervised learning most appropriate?
easy
A. When you want to cluster similar data points without labels.
B. When you have labeled data and want to predict outcomes.
C. When you need to explore an environment to maximize rewards.
D. When you want to reduce dimensionality of data without labels.

Solution

  1. Step 1: Identify the scenario.

    Supervised learning is used when labeled data is available and the goal is to predict outcomes.

  2. Step 2: Eliminate wrong options.

    When you want to cluster similar data points without labels. describes unsupervised learning (clustering), When you need to explore an environment to maximize rewards. describes reinforcement learning (exploration and rewards), and When you want to reduce dimensionality of data without labels. describes unsupervised learning (dimensionality reduction).

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Labeled data -> Supervised learning [OK]
Hint: Labeled data -> Supervised learning [OK]
Common Mistakes:
4. When dealing with high-dimensional data, which approach is more suitable for reducing dimensionality while preserving variance?
medium
A. Use k-means clustering to group similar data points.
B. Use L1 regularization to select features.
C. Apply PCA to project data onto a lower-dimensional space.
D. Apply dropout to randomly ignore features during training.

Solution

  1. Step 1: Identify the goal of preserving variance.

    PCA is designed to reduce dimensions while preserving variance.

  2. Step 2: Eliminate wrong options.

    Use L1 regularization to select features. selects features, not reduce dimensions. Use k-means clustering to group similar data points. clusters data, not reduce dimensions. Apply dropout to randomly ignore features during training. is for regularization, not dimensionality reduction.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    PCA reduces dimensions, keeps variance. [OK]
Hint: PCA reduces dimensions, keeps variance. [OK]
Common Mistakes:
5. You need to deploy an object detection model on a mobile device with limited computational resources. Which approach would you take?
hard
A. Use a lightweight version of YOLO, like Tiny YOLO
B. Use a full YOLO model for its speed
C. Use a full R-CNN model for its accuracy
D. Use a lightweight version of R-CNN

Solution

  1. Step 1: Consider resource constraints

    Mobile devices require models that are both fast and lightweight.

  2. Step 2: Match model to constraints

    Tiny YOLO is designed for environments with limited resources, balancing speed and size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Mobile = Tiny YOLO [OK]
Hint: Mobile = Tiny YOLO [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard