Bird
Raised Fist0

Semantic Search - Dense Retrieval, Bi-Encoder, Cross-Encoder

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Semantic Search - Dense Retrieval, Bi-Encoder, Cross-Encoder
Design a semantic search system using dense retrieval with bi-encoder and cross-encoder models.
system-designhardFreq: very_highRanking & Retrieval
Prerequisites:EmbeddingsTransformer ArchitectureVector Search
🎯
Problem Framing
Business Objective

Maximize user satisfaction by providing the most relevant search results, thereby increasing user retention and engagement.

ML Objective

Maximize the relevance score of retrieved documents given a user query using dense retrieval techniques.

Constraints
  • P99 latency < 100ms for query response
  • High throughput of 1000 QPS
  • Cold start problem for new documents
  • Limited labeled data for training
  • Compliance with data privacy regulations
Clarifying Questions
  • What is the expected scale of the document corpus?
  • How frequently do new documents get added to the corpus?
  • What are the primary user demographics and their search behavior?
  • Are there any specific compliance requirements for data handling?
  • What is the acceptable trade-off between latency and accuracy?
Out of Scope
✗ UI/UX design of the search interface
✗ Backend infrastructure unrelated to ML
✗ Non-ML based search optimizations
📊
Offline Metrics
NDCG@10 because it weights top results more heavily, aligning with user satisfaction
Recall@100 to ensure a broad set of relevant documents are retrieved initially
📈
Online Metrics
CTR (Click-Through Rate) to measure user engagement with search results
Session duration as a proxy for user satisfaction with search results
⚖️
Trade-offs

High offline NDCG might not always translate to high online CTR if user behavior changes or if the model overfits to training data.

💼
Business Metric

User retention rate, as it directly correlates with long-term engagement and revenue.

🗄️
Data Sources
  • User query logs (text, real-time)
  • Document corpus (text, updated daily)
  • User interaction data (clicks, dwell time, real-time)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user clicks and dwell time, as explicit labels are scarce and costly.

Data Volume

Approximately 10 million queries per day, 1 TB of new documents per month

Freshness

Features should be updated in real-time to T-1 day to capture the latest user interactions.

⚠️
Data Challenges
  • Sparse labels due to implicit feedback
  • Position bias in click data
  • Cold start for new documents with no interaction history
  • Handling diverse query intents
🔧
Key Features
embedding
Query embedding
Captures semantic meaning of user queries
embedding
Document embedding
Represents documents in the same semantic space as queries
categorical
User click history
Personalizes search results based on past behavior
numerical
Dwell time
Indicates user satisfaction with clicked results
numerical
Query length
Helps in understanding query complexity
⚡
Feature Engineering Notes
Real-time vs Batch

Query embeddings need real-time computation, while document embeddings can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_query_embedding(query, model):
    return model.encode(query)
🤖
Model Selection
Baseline

TF-IDF based retrieval

Proposed Model

Bi-Encoder for retrieval followed by Cross-Encoder for re-ranking

Bi-Encoder allows for efficient retrieval by precomputing document embeddings, while Cross-Encoder improves ranking accuracy by considering query-document interactions.
✗
Alternatives Rejected
XGBoost: Rejected due to inability to handle high-dimensional embeddings efficiently.
Single-stage Transformer: Rejected due to high latency in processing large document sets.
🏋️
Training Strategy

Offline training with periodic updates to incorporate new data and adapt to changing user behavior.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput1000 QPS
ArchitectureTwo-stage: Bi-Encoder for initial retrieval and Cross-Encoder for re-ranking to balance efficiency and accuracy.
CachingCache top N results for popular queries with a TTL of 1 hour to reduce computation load.
FallbackUse cached results or a rule-based system if the model is unavailable.
📡
Data Drift Signals
Shift in query distribution
Change in document feature distributions
Decrease in click-through rates
📉
Model Drift Signals
Drop in NDCG@10
Increase in prediction variance
Shift in embedding space distributions
🔔
Alerting & Retraining
Retraining Trigger

Retrain when NDCG@10 drops by more than 5% over a month.

Alerting Strategy

On-call engineer paged for significant drops in CTR or NDCG, with logs for query latency and error rates.

🎯
Interview Angles
  • How do you handle the cold start problem for new documents?
  • What trade-offs do you consider between latency and accuracy?
  • How do you ensure the system adapts to changing user behavior?
⚠️
Common Mistakes
Ignoring the impact of position bias in click data
Overlooking the need for real-time feature updates
Choosing a model that doesn't meet latency constraints
Failing to consider the scalability of the retrieval system
📌
Key Decisions
  • Choosing a two-stage architecture to balance efficiency and accuracy
  • Using implicit feedback for labeling due to sparse explicit labels
  • Selecting Bi-Encoder and Cross-Encoder models to meet latency and accuracy requirements

Practice

(1/5)
1. In an A/B testing platform for ML models, which component is responsible for ensuring that users are consistently assigned to the same variant across multiple sessions?
easy
A. Randomization Engine
B. User Segmentation Module
C. Experiment Dashboard
D. Session Persistence Layer

Solution

  1. Step 1: Identify the role of each component.

    The Session Persistence Layer ensures consistent user assignment across sessions.

  2. Step 2: Eliminate wrong options.

    Randomization Engine assigns variants initially, User Segmentation Module groups users, and Experiment Dashboard displays results.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Session consistency needs persistence. [OK]
Hint: Session consistency needs persistence. [OK]
Common Mistakes:
2. For a learning to rank system, which metric is most appropriate to evaluate the quality of the ranking?
easy
A. Normalized Discounted Cumulative Gain (NDCG)
B. Mean Squared Error (MSE)
C. Precision
D. Recall

Solution

  1. Step 1: Identify the metric that evaluates ranking quality.

    NDCG is specifically designed to evaluate the quality of rankings.

  2. Step 2: Eliminate wrong options.

    MSE is for regression, precision and recall are for classification, not ranking.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    NDCG = ranking quality [OK]
Hint: NDCG = ranking quality [OK]
Common Mistakes:
3. A delivery forecasting model is consistently overestimating ETAs during peak traffic hours. What is the most likely failure mode?
medium
A. The model is not accounting for real-time traffic data
B. The model has too many parameters
C. The model is underfitting the data
D. The model is overfitting to the training data

Solution

  1. Step 1: Identify the cause of overestimation during peak hours

    Real-time traffic data is crucial for accurate ETA predictions.

  2. Step 2: Eliminate wrong options

    Underfitting (A) and overfitting (D) are general issues, not specific to peak hours. Too many parameters (C) would cause overfitting, not specific time errors.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time data is key for peak accuracy. [OK]
Hint: Real-time data is key for peak accuracy. [OK]
Common Mistakes:
4. In a feed ranking system, what is a likely failure mode if the system does not update user preferences frequently?
medium
A. Increased server load
B. Higher click-through rates
C. Decreased user engagement
D. Improved content diversity

Solution

  1. Step 1: Consider the impact of stale preferences

    If user preferences are not updated, the system may show irrelevant content.

  2. Step 2: Eliminate wrong options

    Stale preferences don't increase server load or improve diversity. They decrease engagement, not increase CTR.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stale data reduces engagement. [OK]
Hint: Stale data reduces engagement. [OK]
Common Mistakes:
5. In a two-stage retrieval system, when should you prioritize a simpler candidate generation model over a complex one?
medium
A. When the ranking stage is computationally expensive and needs more resources.
B. When precision is more important than recall in the candidate generation stage.
C. When the dataset is small and computation resources are abundant.
D. When the ranking model is not capable of handling large candidate sets.

Solution

  1. Step 1: Consider resource allocation

    If the ranking stage is computationally expensive, a simpler candidate generation model can save resources for the ranking stage.

  2. Step 2: Eliminate wrong options

    When the dataset is small and computation resources are abundant. is incorrect because a small dataset doesn't necessitate a simpler model. When precision is more important than recall in the candidate generation stage. is incorrect because recall is typically prioritized in candidate generation. When the ranking model is not capable of handling large candidate sets. is incorrect because the ranking model should handle the candidate set size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Save resources for ranking if it's costly. [OK]
Hint: Save resources for ranking if it's costly. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard