Bird
Raised Fist0

Search Ranking System - Query-Document Relevance at Scale

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Design a Scalable Search Ranking System for Query-Document Relevance
45 min · senior
senior

You are tasked with designing a search ranking system for a platform with 50 million daily active users. The system must handle 10,000 queries per second with a P99 latency SLA of 100ms. You have a team of 5 engineers and 8 months to deliver. The data available includes user click-through logs, dwell time, and query-document pairs without explicit relevance labels.

Given Info
  • 50 million daily active users
  • 10,000 queries per second
  • P99 latency SLA of 100ms
  • Team of 5 engineers
  • 8 months to deliver
📋
Guided Steps
1/5
Step 1
Data Collection and Labeling Strategy
How will you collect and label data for training the ranking model given the constraints?
💡 Hint
Consider the types of implicit feedback available and how they can be used to infer relevance.
Key Decision
Use implicit feedback (clicks, dwell time) to infer relevance labels because explicit labels are unavailable and costly to obtain at this scale.
Interviewer Expectation
Mid: Proposes using click-through data for labeling. Senior: Also considers dwell time as a signal and discusses potential biases in implicit feedback.
⚠️Common Wrong Answer
Relying solely on click data without considering its biases, leading to a model that overfits to clickbait.
Step 2
Model Selection
What model architecture will you choose for ranking, and why?
💡 Hint
Think about the trade-off between model complexity and inference latency.
Key Decision
Choose a two-tower neural network model because it allows pre-computation of document embeddings, reducing online latency.
Interviewer Expectation
Mid: Suggests a gradient boosting model. Senior: Justifies a two-tower model for its scalability and latency benefits, despite higher training costs.
⚠️Common Wrong Answer
Selecting a complex deep learning model without considering inference latency, leading to SLA violations.
Step 3
Feature Engineering
What features will you use, and how will you ensure they are available at inference time?
💡 Hint
Consider the balance between feature richness and real-time availability.
Key Decision
Use features that can be pre-computed or are available in real-time to meet the latency SLA.
Interviewer Expectation
Mid: Lists basic features like term frequency. Senior: Discusses advanced features like user and document embeddings and their pre-computation.
⚠️Common Wrong Answer
Using features that require real-time computation, causing latency issues.
Step 4
Serving Architecture
How will you design the serving architecture to meet the latency requirements?
💡 Hint
Think about caching strategies and the separation of online and offline computations.
Key Decision
Implement a caching layer for pre-computed embeddings and use a fast retrieval system to meet the 100ms SLA.
Interviewer Expectation
Mid: Proposes a basic serving setup. Senior: Details caching strategies and discusses trade-offs between memory usage and latency.
⚠️Common Wrong Answer
Ignoring caching, leading to high latency due to repeated computations.
Step 5
Monitoring and Evaluation
How will you monitor the system and evaluate its performance in production?
💡 Hint
Consider both offline and online metrics and how they relate to business objectives.
Key Decision
Monitor CTR and latency in production while using offline metrics like NDCG@10 to evaluate model quality.
Interviewer Expectation
Mid: Mentions basic monitoring. Senior: Connects metrics to business objectives and discusses alerting for drift and latency issues.
⚠️Common Wrong Answer
Focusing only on offline metrics, missing real-time performance issues.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Uses implicit feedback for labeling due to lack of explicit labels.
  • Chooses a model architecture that balances complexity with latency constraints.
  • Designs a serving architecture that meets the 100ms latency SLA.
  • Distinguishes between offline and online metrics and explains their relevance.
Good to Have - 30%
  • Considers biases in implicit feedback and proposes mitigation strategies.
  • Discusses advanced feature engineering techniques and their impact on latency.
  • Proposes a comprehensive monitoring strategy that includes drift detection.
  • Explains trade-offs between model complexity and serving latency.
Bonus - 10%
  • Quantifies the impact of caching strategies on latency.
  • Discusses the scalability of the chosen model architecture.
  • Proposes a plan for continuous model improvement and retraining.
✅
Model Answer
▼ Reveal

The ideal solution involves using implicit feedback to infer relevance labels, selecting a two-tower neural network for its scalability and latency benefits, and designing a serving architecture with caching to meet the 100ms SLA. Monitoring should focus on both CTR and latency, with offline metrics like NDCG@10 to evaluate model quality.

Key Decisions
  • Use implicit feedback (clicks, dwell time) for labeling due to lack of explicit labels.
  • Select a two-tower model to pre-compute document embeddings, reducing online latency.
  • Implement a caching layer to store pre-computed embeddings, ensuring fast retrieval.
  • Monitor CTR and latency in production, using offline metrics like NDCG@10 for evaluation.
Anti-patterns
Choosing a complex model without considering inference latency.
Ignoring the need for caching, leading to high latency.
Focusing solely on offline metrics, missing real-time performance issues.

Practice

(1/5)
1. Which metric is most appropriate for evaluating the performance of a Two-Tower Model for Retrieval?
easy
A. Mean Squared Error (MSE)
B. Log Loss
C. F1 Score
D. Precision at K

Solution

  1. Step 1: Identify the goal of the retrieval model

    The goal is to rank items effectively for users, making ranking metrics like Precision at K suitable.

  2. Step 2: Eliminate wrong options

    Mean Squared Error (MSE) (MSE) is for regression tasks. F1 Score (F1 Score) is for classification tasks. Log Loss (Log Loss) is also for classification tasks.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use ranking metrics like Precision at K. [OK]
Hint: Use ranking metrics like Precision at K. [OK]
Common Mistakes:
2. When choosing between behavioral signals and device fingerprinting for bot detection, which is more suitable for detecting sophisticated bots and why?
medium
A. Behavioral signals, because they are harder for bots to mimic.
B. Device fingerprinting, because it provides a unique identifier.
C. Behavioral signals, because they are easier to implement.
D. Device fingerprinting, because it is less prone to false positives.

Solution

  1. Step 1: Consider sophistication of bots.

    Sophisticated bots can mimic device fingerprints but struggle with human-like behavior.

  2. Step 2: Eliminate wrong options.

    Device fingerprinting is easier to spoof; implementation ease is irrelevant to detection capability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sophisticated bots struggle with human behavior [OK]
Hint: Sophisticated bots struggle with human behavior [OK]
Common Mistakes:
3. When deploying a fine-tuning pipeline at scale, should you prioritize using RLHF or DPO for optimizing model performance, and why?
medium
A. Prioritize RLHF because it directly improves model accuracy.
B. Prioritize DPO because it simplifies the model architecture.
C. Prioritize RLHF because it reduces computational costs.
D. Prioritize DPO because it optimizes data quality, which is foundational.

Solution

  1. Step 1: Understand the roles of RLHF and DPO

    RLHF aligns outputs with human preferences, while DPO ensures high-quality data input, which is foundational for any model training.

  2. Step 2: Eliminate wrong options

    A: RLHF improves alignment, not accuracy directly. C: RLHF does not reduce computational costs. D: DPO does not simplify architecture but improves data quality.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Data quality is foundational; prioritize DPO. [OK]
Hint: Data quality is foundational; prioritize DPO. [OK]
Common Mistakes:
4. In a scenario where model accuracy is critical, which approach should be avoided to prevent potential accuracy degradation?
medium
A. KV Cache
B. Speculative Decoding
C. Batching
D. Model Pruning

Solution

  1. Step 1: Understand the impact on accuracy.

    Speculative Decoding can introduce errors by predicting future tokens.

  2. Step 2: Eliminate wrong options.

    Batching and KV Cache do not inherently affect accuracy. Model Pruning reduces model size but can be controlled to maintain accuracy.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Speculative Decoding risks accuracy for speed. [OK]
Hint: Speculative Decoding risks accuracy for speed. [OK]
Common Mistakes:
5. As the user base grows 10x, which component of a hate speech detection system is most likely to become a bottleneck?
hard
A. Model Training
B. Data Preprocessing
C. Feature Extraction
D. Model Inference

Solution

  1. Step 1: Consider the impact of scale

    With a 10x increase in users, the system must handle significantly more inference requests.

  2. Step 2: Eliminate wrong options

    Data preprocessing and feature extraction are typically batch processes, and model training is not affected by user scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    10x users = Inference bottleneck [OK]
Hint: 10x users = Inference bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard