Bird
Raised Fist0

Ad Click-Through Rate Prediction - Logistic Regression to DLRM

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Thursday AUC Drop
Started last Thursday, ongoing for 5 days

CTR prediction model serving 5M ad impressions/day for a major social media platform.

Symptom
Model achieves 0.89 AUC offline but 0.71 AUC on live traffic · Recommendation diversity collapsed, showing repetitive ads · Real-time feature latency spiked from 50ms to 200ms
📡
Signals
metric
Live AUC dropped significantly
Live AUC dropped from 0.85 to 0.71
distribution
Feature distribution shift detected in 'user_activity_level'
Mean shifted from 0.5 to 0.2
latency
Real-time feature extraction latency increased
Latency increased from 50ms to 200ms
feature_importance
Feature 'user_last_active_days' importance dropped
Importance dropped from 0.18 to 0.02
log
Error logs showing increased timeouts in feature service
Timeout errors increased by 300%
🔬
Hypotheses
Feature distribution shift causing model degradation
Supporting Signals
distributionfeature_importance
Verify: Check if recent data distribution matches training data
df['user_activity_level'].hist()
Increased latency causing stale features
Supporting Signals
latencylog
Verify: Check timestamps of feature extraction vs prediction
df['feature_extraction_time'] - df['prediction_time']
🎯
Root Cause
Feature service latency increase causing stale features
Why It Happened

A recent deployment introduced inefficient code in feature service

🔧
Fix Strategy
ImmediateRollback to previous feature service version
Short-termOptimize the new feature service code to reduce latency
Long-termImplement a canary deployment strategy for feature service updates
🐍
Python Fix
Python
def optimize_feature_service():
    # Optimize code here
    pass
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to small training data
Why candidates think this: AUC drop suggests model generalization issue
Why wrong: Validation AUC is stable, indicating training data is sufficient
Feature importance change due to model retraining
Why candidates think this: Feature importance drop suggests model change
Why wrong: No recent model retraining occurred; issue is with feature latency
📊
Monitoring to Add
  • Real-time feature extraction latency alerts
  • Live vs validation AUC discrepancy alerts
  • Feature distribution shift monitoring
📋
Process Change

Require latency benchmarks before deploying feature service updates

🏗️
Architectural Change

Introduce a feature caching layer to handle latency spikes

💡
Key Insight
Latency spikes in feature services can lead to stale data, causing significant model performance degradation.

Practice

(1/5)
1. In the context of avoiding filter bubbles in recommendation systems, which component is responsible for ensuring content diversity?
easy
A. Collaborative filtering
B. Diversity re-ranking layer
C. User profiling
D. Content-based filtering

Solution

  1. Step 1: Identify the role of each component

    Collaborative and content-based filtering focus on user preferences, while diversity re-ranking ensures varied content.

  2. Step 2: Eliminate wrong options

    Collaborative filtering (A) and content-based filtering (B) do not inherently ensure diversity. User profiling (D) focuses on understanding user preferences, not diversity.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Diversity re-ranking ensures varied content [OK]
Hint: Diversity re-ranking ensures varied content [OK]
Common Mistakes:
2. For a real-time recommendation system, which metric should the ML platform optimize to ensure timely and relevant suggestions?
easy
A. Latency
B. Mean Absolute Error (MAE)
C. Precision
D. Recall

Solution

  1. Step 1: Understand the system's requirement.

    Real-time systems need to provide timely responses, making latency crucial.

  2. Step 2: Eliminate wrong options.

    MAE measures error, Precision and Recall focus on accuracy, not speed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs low latency. [OK]
Hint: Real-time systems prioritize low latency. [OK]
Common Mistakes:
3. In a visual search system using image embeddings, which component is responsible for transforming images into a fixed-size vector representation?
easy
A. Approximate Nearest Neighbor (ANN) index
B. Serving layer
C. Image embedding model
D. Database storage

Solution

  1. Step 1: Identify the role of each component

    The image embedding model is responsible for converting images into vector representations.

  2. Step 2: Eliminate wrong options

    ANN index is for searching, serving layer is for delivering results, and database storage is for storing data.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Embeddings come from the embedding model. [OK]
Hint: Embeddings come from the embedding model. [OK]
Common Mistakes:
4. In a RAG system, when should you prefer a dense retriever over a sparse retriever?
medium
A. When the corpus is small and domain-specific
B. When computational resources are limited
C. When interpretability is a priority
D. When the corpus is large and diverse

Solution

  1. Step 1: Understand retriever types.

    Dense retrievers are better for large, diverse corpora as they can capture semantic meaning more effectively.

  2. Step 2: Eliminate wrong options.

    Sparse retrievers are more interpretable and efficient for small, domain-specific corpora.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dense for large, diverse data. [OK]
Hint: Dense for large, diverse data. [OK]
Common Mistakes:
5. As the dataset size increases 10x in a FAISS-based vector search system, which component is most likely to become a bottleneck first?
hard
A. Index storage
B. Search algorithm
C. Dimensionality reduction
D. Quantization process

Solution

  1. Step 1: Consider the impact of dataset size increase.

    Index storage will grow significantly, potentially becoming a bottleneck.

  2. Step 2: Eliminate wrong options.

    Search algorithm, dimensionality reduction, and quantization are less affected by size increase compared to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Index storage grows with dataset size. [OK]
Hint: Index storage grows with dataset size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard