Bird
Raised Fist0

Two-Stage Retrieval - Candidate Generation Then Ranking

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Two-Stage Retrieval - Candidate Generation Then Ranking
Efficiently retrieve and rank items for personalized recommendations.
system-designhardFreq: very_highRanking & Retrieval
Prerequisites:Vector Search and ANNLearning to Rank
🎯
Problem Framing
Business Objective

Maximize user engagement and revenue through personalized recommendations.

ML Objective

Maximize P(click|user, item, context) for recommended items.

Constraints
  • P99 latency < 100ms for end-to-end recommendation
  • High throughput: 10,000 QPS
  • Cold start for new users and items
  • Data privacy regulations (GDPR compliance)
  • Limited labeled data for new items
Clarifying Questions
  • What is the expected scale of items and users?
  • How frequently do new items get added to the catalog?
  • What is the acceptable trade-off between latency and accuracy?
  • Are there any specific privacy constraints we need to consider?
  • How do we handle recommendations for users with sparse interaction history?
Out of Scope
✗ User interface design
✗ Backend infrastructure scaling
✗ Non-personalized recommendations
📊
Offline Metrics
PR-AUC: Suitable for imbalanced click data where positive interactions are rare.
NDCG@10: Focuses on the quality of the top-ranked items, which is crucial for user satisfaction.
📈
Online Metrics
CTR: Directly measures user engagement with recommendations.
Revenue per session: Captures the business impact of recommendations.
⚖️
Trade-offs

Offline metrics like PR-AUC may not always correlate with online metrics like CTR due to differences in user behavior in live environments.

💼
Business Metric

Overall revenue generated from recommendations.

🗄️
Data Sources
  • User interaction logs (clicks, views) - real-time
  • Item catalog metadata (descriptions, categories) - daily updates
  • User profiles (demographics, preferences) - weekly updates
  • External data (trending topics, seasonal effects) - daily updates
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, views) as labels for training.

Data Volume

Approximately 100 million interactions per day, 10 TB/month.

Freshness

Real-time updates for user interactions, T-1 day for item metadata.

⚠️
Data Challenges
  • Sparse labels for new items
  • Position bias in interaction logs
  • Survivorship bias in user data
  • Cold start for new users and items
🔧
Key Features
embedding
User embedding
Captures user preferences and history.
embedding
Item embedding
Encodes item characteristics and popularity.
numerical
User-item interaction history
Reflects past user behavior with items.
categorical
Time of day
Accounts for temporal patterns in user activity.
categorical
Device type
Different devices may influence interaction patterns.
⚡
Feature Engineering Notes
Real-time vs Batch

User-item interaction history and embeddings need real-time updates, while item metadata can be batch processed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_user_embedding(user_id, interaction_history):
    # Example function to compute user embedding
    return np.mean([item_embedding[item] for item in interaction_history], axis=0)
🤖
Model Selection
Baseline

Popularity-based model

Proposed Model

Two-tower neural network model

The two-tower model allows for efficient precomputation of item embeddings, reducing online latency to a single dot product lookup.
✗
Alternatives Rejected
Matrix Factorization: Rejected due to inability to handle new items without retraining.
XGBoost: Rejected due to high latency and memory overhead incompatible with real-time requirements.
🏋️
Training Strategy

Offline training with periodic updates to incorporate new data and retrain embeddings.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage architecture with candidate generation followed by ranking to balance efficiency and accuracy.
CachingCache popular item embeddings with a TTL of 1 hour, invalidate on item updates.
FallbackUse rule-based recommendations or cached results if the model is unavailable.
📡
Data Drift Signals
Shift in user interaction distribution
Change in item popularity trends
New user demographics appearing
📉
Model Drift Signals
Decrease in PR-AUC over time
Shift in prediction score distribution
Increase in prediction latency
🔔
Alerting & Retraining
Retraining Trigger

PR-AUC drops below 0.75 or significant drift in user interaction patterns.

Alerting Strategy

On-call engineer paged for significant metric drops, logs monitored for latency spikes, auto-recovery for minor prediction failures.

🎯
Interview Angles
  • How do you handle cold start problems in this system?
  • What are the trade-offs between latency and accuracy in your design?
  • How do you ensure data privacy and compliance with regulations?
⚠️
Common Mistakes
Ignoring the latency constraints in model selection.
Overlooking the importance of real-time feature updates.
Failing to address cold start issues for new users and items.
Not considering the impact of data drift on model performance.
Choosing complex models without justifying their necessity.
📌
Key Decisions
  • Choosing a two-tower model for efficient embedding computation.
  • Balancing real-time and batch processing for feature updates.
  • Implementing a robust caching strategy to meet latency requirements.

Practice

(1/5)
1. For an LLM serving system focused on minimizing response time, which metric should be prioritized?
easy
A. Latency
B. Memory Usage
C. Accuracy
D. Throughput

Solution

  1. Step 1: Identify the goal.

    Minimizing response time directly relates to reducing latency.

  2. Step 2: Eliminate wrong options.

    Throughput measures volume, not speed. Accuracy is about correctness, not speed. Memory Usage affects capacity, not response time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Minimize latency to cut response time. [OK]
Hint: Minimize latency to cut response time. [OK]
Common Mistakes:
2. In the context of an ML monitoring platform, which component is primarily responsible for identifying when the input data distribution has changed significantly?
easy
A. Alerting System
B. Retraining Scheduler
C. Model Evaluation Component
D. Drift Detection Module

Solution

  1. Step 1: Identify the role of each component

    The Drift Detection Module is specifically designed to monitor changes in data distribution.

  2. Step 2: Eliminate wrong options

    Alerting System notifies about issues, Retraining Scheduler handles model updates, and Model Evaluation Component assesses model performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Drift Detection = Data Change [OK]
Hint: Drift Detection = Data Change [OK]
Common Mistakes:
3. In an ML monitoring platform, what is a likely failure mode if the alerting system is not properly calibrated?
medium
A. The retraining process will be triggered too frequently
B. The system will miss significant drifts
C. The model's accuracy will degrade over time
D. The model will overfit to the training data

Solution

  1. Step 1: Understand the role of alerting calibration

    Proper calibration ensures alerts are meaningful and not overly sensitive.

  2. Step 2: Identify the failure mode

    If not calibrated, the alerting system may trigger retraining too often, leading to unnecessary computational costs.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Over-alerting = Frequent retraining [OK]
Hint: Over-alerting = Frequent retraining [OK]
Common Mistakes:
4. When choosing between a Bi-Encoder and a Cross-Encoder for a semantic search task, which is more suitable for real-time retrieval and why?
medium
A. Bi-Encoder, because it allows pre-computation of document embeddings
B. Cross-Encoder, because it can handle large datasets efficiently
C. Cross-Encoder, because it provides higher accuracy by evaluating pairs
D. Bi-Encoder, because it requires less computational power during retrieval

Solution

  1. Step 1: Compare Bi-Encoder and Cross-Encoder

    Bi-Encoders allow pre-computation of document embeddings, making them suitable for real-time retrieval.

  2. Step 2: Eliminate wrong options

    Cross-Encoders evaluate pairs at query time, which is computationally expensive and not suitable for real-time retrieval.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bi-Encoder = pre-compute embeddings [check]
Hint: Bi-Encoder = pre-compute embeddings [check]
Common Mistakes:
5. As the user base of an ML platform grows 10x, which component is most likely to become a bottleneck first?
hard
A. Model Training Pipeline
B. Feature Store
C. Serving Infrastructure
D. Data Ingestion Pipeline

Solution

  1. Step 1: Consider the impact of scaling on each component.

    Serving Infrastructure is directly impacted by increased user requests, making it a likely bottleneck.

  2. Step 2: Eliminate wrong options.

    Model Training and Data Ingestion are less affected by user growth, and Feature Store scales with data, not users.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Serving scales with user requests. [OK]
Hint: Serving scales with user requests. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard