Bird
Raised Fist0

Search Ranking System - Query-Document Relevance at Scale

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Search Ranking System - Query-Document Relevance at Scale
Design a scalable search ranking system optimizing query-document relevance.
system-designhardFreq: very_highRanking & Retrieval
Prerequisites:Learning to RankTwo-Stage RetrievalRanking Metrics - NDCG, MAP, MRR
🎯
Problem Framing
Business Objective

Maximize user engagement by improving the relevance of search results.

ML Objective

Maximize NDCG@10 for search results to ensure top results are most relevant.

Constraints
  • P99 latency < 100ms for query response
  • High throughput: 10,000 QPS
  • Cold start for new documents
  • Regulatory compliance for data usage
  • Limited labeled data for new queries
Clarifying Questions
  • What is the current baseline performance in terms of NDCG?
  • How frequently do new documents get added to the index?
  • What is the acceptable trade-off between latency and relevance?
  • Are there specific user segments we need to prioritize?
  • What are the primary sources of user feedback for relevance?
Out of Scope
✗ UI/UX design of the search interface
✗ Backend infrastructure scaling
✗ Non-relevance based ranking factors like ads
📊
Offline Metrics
NDCG@10: Focuses on the top 10 results, ensuring high relevance where users are most likely to click.
MAP: Measures precision across all relevant documents, useful for overall relevance assessment.
📈
Online Metrics
CTR: Directly measures user engagement with search results.
Session length: Indicates user satisfaction with search results.
⚖️
Trade-offs

Offline metrics like NDCG@10 may show improvement, but if CTR doesn't increase, it indicates a mismatch between predicted relevance and user perception.

💼
Business Metric

Overall user engagement measured by CTR and session length.

🗄️
Data Sources
  • Search query logs: Real-time, capturing user interactions.
  • Document metadata: Updated daily, includes titles, descriptions.
  • User feedback: Implicit feedback from clicks, explicit ratings.
  • External knowledge bases: Weekly updates, enrich document context.
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user clicks and dwell time, as explicit labels are sparse and costly.

Data Volume

Approximately 100 million queries per day, 10 TB of data per month.

Freshness

Features should be updated in near real-time to reflect the latest user interactions.

⚠️
Data Challenges
  • Sparse labels for new queries
  • Position bias in click data
  • Survivorship bias from popular documents
  • Cold start for new documents
🔧
Key Features
numerical
Query-Document Similarity
Measures relevance directly, crucial for ranking.
embedding
User Click History
Captures user preferences and personalization.
numerical
Document Freshness
Recent documents may be more relevant.
numerical
Query Length
Longer queries may indicate more specific intent.
categorical
User Location
Geographical relevance can impact result ranking.
⚡
Feature Engineering Notes
Real-time vs Batch

Query-Document Similarity and User Click History need real-time computation, while Document Freshness and User Location can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_similarity(query, document):
    return cosine_similarity(query.vector, document.vector)
🤖
Model Selection
Baseline

BM25 for initial relevance scoring.

Proposed Model

Two-tower neural network model.

Allows precomputing document embeddings offline, reducing online latency to a single dot product lookup.
✗
Alternatives Rejected
XGBoost: Rejected due to inability to handle sparse high-dimensional data efficiently.
Single-tower neural network: Rejected due to higher latency in computing joint embeddings online.
🏋️
Training Strategy

Offline training with periodic updates using new query-document interactions.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage retrieval and ranking to balance efficiency and relevance.
CachingCache popular query results with a TTL of 5 minutes, invalidate on document updates.
FallbackUse BM25-based retrieval if the model is unavailable.
📡
Data Drift Signals
Shift in query distribution
Change in document feature distributions
Decrease in click-through rates
📉
Model Drift Signals
NDCG degradation over time
Increase in prediction variance
Drop in MAP scores
🔔
Alerting & Retraining
Retraining Trigger

NDCG@10 drops below 0.8 for a sustained period of 1 week.

Alerting Strategy

PagerDuty alerts for significant metric drops, automated logs for data drift, auto-recovery scripts for minor issues.

🎯
Interview Angles
  • How do you handle cold start for new documents?
  • What trade-offs do you make between latency and relevance?
  • How do you ensure the model adapts to changing user behavior?
⚠️
Common Mistakes
Ignoring latency constraints in model selection.
Overfitting to offline metrics without considering online performance.
Neglecting the impact of data freshness on relevance.
📌
Key Decisions
  • Choosing a two-tower model for latency efficiency.
  • Balancing real-time and batch feature computation.
  • Implementing a robust monitoring and retraining strategy.

Practice

(1/5)
1. When deciding between a full factorial design and a fractional factorial design for an A/B test, which is more appropriate when the number of factors is large and why?
medium
A. Full factorial design, because it tests all possible combinations.
B. Fractional factorial design, because it is easier to interpret.
C. Fractional factorial design, because it reduces the number of tests needed.
D. Full factorial design, because it provides more accurate results.

Solution

  1. Step 1: Consider the number of factors.

    With a large number of factors, testing all combinations becomes impractical.

  2. Step 2: Eliminate wrong options.

    Full factorial is impractical for large factors, and fractional factorial is not necessarily easier to interpret.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Large factors need fewer tests. [OK]
Hint: Large factors need fewer tests. [OK]
Common Mistakes:
2. When choosing between pairwise and listwise approaches for a ranking task, which scenario favors the listwise approach?
medium
A. When the ranking task requires optimizing the entire list's order
B. When the ranking task involves a large number of items
C. When computational resources are limited
D. When the goal is to optimize for individual item relevance

Solution

  1. Step 1: Identify the scenario where listwise is beneficial.

    Listwise approaches are best when optimizing the entire list's order is crucial.

  2. Step 2: Eliminate wrong options.

    Pairwise is better for individual relevance, and listwise can be resource-intensive, making A and B incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Listwise = full list optimization [OK]
Hint: Listwise = full list optimization [OK]
Common Mistakes:
3. In an ML monitoring platform, when should you prioritize retraining the model over just alerting the team?
medium
A. When there's a significant drift detected in input data
B. When the model's accuracy drops slightly
C. When the model's predictions are consistent but incorrect
D. When the alerting system triggers multiple false positives

Solution

  1. Step 1: Identify scenarios for retraining

    Retraining is crucial when there's a significant drift in input data, indicating the model may no longer be valid.

  2. Step 2: Eliminate wrong options

    Minor accuracy drops or consistent incorrect predictions may not require retraining. False positives in alerts suggest tuning the alert system, not retraining.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Significant drift = Retrain [OK]
Hint: Significant drift = Retrain [OK]
Common Mistakes:
4. In a multimodal system, if the model's performance suddenly drops after integrating a new dataset, what is the most likely failure mode?
medium
A. Domain mismatch between new and existing datasets
B. Data leakage between training and validation sets
C. Overfitting due to increased model complexity
D. Insufficient training epochs for the new dataset

Solution

  1. Step 1: Analyze the impact of new data

    Integrating a new dataset can introduce domain differences that the model isn't trained to handle.

  2. Step 2: Identify the most likely cause

    Domain mismatch is a common issue when adding new datasets, leading to performance drops.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    New data can cause domain mismatch issues. [OK]
Hint: New data can cause domain mismatch issues. [OK]
Common Mistakes:
5. As the user base grows 10x, which component of a hate speech detection system is most likely to become a bottleneck?
hard
A. Model Training
B. Data Preprocessing
C. Feature Extraction
D. Model Inference

Solution

  1. Step 1: Consider the impact of scale

    With a 10x increase in users, the system must handle significantly more inference requests.

  2. Step 2: Eliminate wrong options

    Data preprocessing and feature extraction are typically batch processes, and model training is not affected by user scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    10x users = Inference bottleneck [OK]
Hint: 10x users = Inference bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard