Bird
Raised Fist0

Learning to Rank - Pointwise, Pairwise, Listwise Approaches

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Ranking Reversal
Started two weeks ago, ongoing

A learning-to-rank model for a large e-commerce platform, serving personalized product recommendations to 5M users daily.

Symptom
AUC dropped from 0.85 to 0.72 over the past two weeks · Recommendation diversity collapsed, with many users seeing similar products · Real-time feature latency spiked from 50ms to 200ms · A/B test shows offline metrics are stable but online metrics have degraded
📡
Signals
metric
Live AUC dropped significantly
AUC dropped from 0.85 to 0.72
distribution
Recommendation diversity collapsed
Diversity index dropped from 0.65 to 0.30
latency
Real-time feature extraction latency increased
Latency spiked from 50ms to 200ms
feature_importance
Feature 'user_last_active_days' importance changed
Importance dropped from 0.18 to 0.02
metric
Offline AUC remains stable
Offline AUC stable at 0.89
🔬
Hypotheses
Feature extraction latency is causing stale features to be used in real-time predictions.
Supporting Signals
Real-time feature latency spiked from 50ms to 200ms
Verify: Check timestamps of features used in predictions to see if they are outdated.
import pandas as pd

# Assuming df is the DataFrame with prediction logs
df['feature_timestamp_diff'] = (pd.to_datetime(df['prediction_time']) - pd.to_datetime(df['feature_timestamp'])).dt.total_seconds()
print(df['feature_timestamp_diff'].describe())
A recent deployment changed feature importance, affecting model performance.
Supporting Signals
Feature 'user_last_active_days' importance changed
Verify: Compare feature importances before and after the deployment.
import matplotlib.pyplot as plt

# Assuming feature_importance_df is a DataFrame with feature importances
feature_importance_df.plot(kind='bar')
plt.title('Feature Importance Before and After Deployment')
plt.show()
Data distribution shift in live traffic not reflected in offline validation.
Supporting Signals
AUC dropped from 0.85 to 0.72Offline AUC stable at 0.89
Verify: Analyze live traffic data distribution and compare with training data.
import seaborn as sns
import matplotlib.pyplot as plt

# Assuming live_data and train_data are DataFrames
sns.kdeplot(live_data['feature'], label='Live')
sns.kdeplot(train_data['feature'], label='Train')
plt.legend()
plt.show()
🎯
Root Cause
Feature extraction latency caused stale features to be used in real-time predictions, leading to degraded model performance.
Why It Happened

A recent infrastructure change increased latency in the feature extraction pipeline, causing outdated features to be used in predictions.

🔧
Fix Strategy
ImmediateRollback to the previous infrastructure setup to reduce feature extraction latency.
Short-termOptimize the feature extraction pipeline to ensure real-time features are fresh.
Long-termImplement a monitoring system to alert on feature freshness and latency issues.
🐍
Python Fix
Python
# Example code to optimize feature extraction pipeline

def optimize_feature_extraction(data):
    # Simulate optimization by reducing processing time
    import time
    time.sleep(0.05)  # Reduced from 0.2 seconds
    return data

# Apply optimization
optimized_data = optimize_feature_extraction(raw_data)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The stable offline AUC suggests the model is fine, leading to assumptions about overfitting.
Why wrong: The issue is with real-time feature freshness, not model overfitting.
A bug in the recommendation logic is causing diversity collapse.
Why candidates think this: The diversity collapse suggests a logic error in recommendations.
Why wrong: The diversity issue is a symptom of stale features, not a logic bug.
📊
Monitoring to Add
  • Monitor feature extraction latency and alert if it exceeds 100ms
  • Track feature freshness by comparing feature timestamps with prediction times
  • Set up alerts for significant drops in recommendation diversity
  • Monitor real-time AUC and compare with offline AUC regularly
📋
Process Change

Implement a pre-deployment checklist that includes latency and feature freshness tests.

🏗️
Architectural Change

Design the system to decouple feature extraction from real-time predictions, allowing for asynchronous updates.

💡
Key Insight
Feature freshness is critical in real-time systems; latency spikes can silently degrade model performance.

Practice

(1/5)
1. In a spam detection system, which feature is most likely to be effective in identifying spam emails?
easy
A. Email length
B. Presence of certain keywords
C. Sender's email domain
D. Time of day email was sent

Solution

  1. Step 1: Identify effective features for spam detection

    Keywords like 'free', 'win', or 'prize' are often used in spam emails.

  2. Step 2: Eliminate less effective features

    Email length (A) and time of day (D) are not strong indicators of spam. Sender's domain (C) can be spoofed.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Keywords often signal spam. [OK]
Hint: Keywords often signal spam. [OK]
Common Mistakes:
2. When choosing between Isolation Forest and Autoencoder for anomaly detection, which scenario favors using an Autoencoder?
medium
A. When the dataset is small and labeled.
B. When interpretability of the model is crucial.
C. When the anomalies are expected to be in high-dimensional data.
D. When the dataset is large and unlabeled.

Solution

  1. Step 1: Consider the strengths of Autoencoders.

    Autoencoders are particularly effective for high-dimensional data due to their ability to learn complex representations.

  2. Step 2: Eliminate wrong options.

    When the dataset is small and labeled. is incorrect because Autoencoders require more data to train effectively. When interpretability of the model is crucial. is incorrect because Autoencoders are less interpretable than Isolation Forests. When the dataset is large and unlabeled. is incorrect because Autoencoders are computationally intensive and may not be ideal for very large datasets.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Autoencoder = high-dimensional data strength [OK]
Hint: Autoencoder = high-dimensional data strength [OK]
Common Mistakes:
3. When choosing between a Bi-Encoder and a Cross-Encoder for a semantic search task, which is more suitable for real-time retrieval and why?
medium
A. Bi-Encoder, because it allows pre-computation of document embeddings
B. Cross-Encoder, because it can handle large datasets efficiently
C. Cross-Encoder, because it provides higher accuracy by evaluating pairs
D. Bi-Encoder, because it requires less computational power during retrieval

Solution

  1. Step 1: Compare Bi-Encoder and Cross-Encoder

    Bi-Encoders allow pre-computation of document embeddings, making them suitable for real-time retrieval.

  2. Step 2: Eliminate wrong options

    Cross-Encoders evaluate pairs at query time, which is computationally expensive and not suitable for real-time retrieval.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bi-Encoder = pre-compute embeddings [check]
Hint: Bi-Encoder = pre-compute embeddings [check]
Common Mistakes:
4. As the number of users and items increases in a session-based recommendation system, which component is likely to become a bottleneck first?
hard
A. Model training time
B. Data storage capacity
C. Real-time inference latency
D. Feature extraction pipeline

Solution

  1. Step 1: Consider scaling impacts

    Real-time inference latency often becomes a bottleneck as the system scales, due to increased computational demands.

  2. Step 2: Eliminate less likely bottlenecks

    Model training time and data storage can be managed with batch processing and scalable storage solutions; feature extraction is typically pre-computed.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Inference latency scales poorly. [OK]
Hint: Inference latency scales poorly. [OK]
Common Mistakes:
5. As the dataset size increases 10x in a FAISS-based vector search system, which component is most likely to become a bottleneck first?
hard
A. Index storage
B. Search algorithm
C. Dimensionality reduction
D. Quantization process

Solution

  1. Step 1: Consider the impact of dataset size increase.

    Index storage will grow significantly, potentially becoming a bottleneck.

  2. Step 2: Eliminate wrong options.

    Search algorithm, dimensionality reduction, and quantization are less affected by size increase compared to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Index storage grows with dataset size. [OK]
Hint: Index storage grows with dataset size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard