Bird
Raised Fist0

Semantic Search - Dense Retrieval, Bi-Encoder, Cross-Encoder

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Bi-Encoder vs Cross-Encoder

In a semantic search system with a 100ms latency SLA and a dataset of 1 million documents, would you use a Bi-Encoder or a Cross-Encoder? Justify your choice.

📊
Comparison
DimensionBi-EncoderCross-Encoder
AccuracyMediumHigh
Inference LatencyLow (<100ms)High (>100ms)
ScalabilityHighLow
Compute CostLowHigh
Interaction ModelingLowHigh
✅
Verdict
Default to Bi-Encoders for large-scale retrieval tasks with strict latency requirements; switch to Cross-Encoders when accuracy is critical and dataset size is manageable.
💡
Key Insight
Bi-Encoders excel in scalability and speed, while Cross-Encoders provide superior accuracy through interaction modeling.
⚠️
Interview Trap
❌ Wrong: Always use Cross-Encoders for better accuracy.
Ignoring scalability and latency constraints for large datasets.
✅ Consider the trade-off between accuracy and scalability; use Bi-Encoders for large datasets with strict latency requirements and Cross-Encoders for smaller datasets where accuracy is paramount.
Bi-Encoder
"High scalability for large datasets due to independent encoding of queries and documents."
Weaknesses
  • Lower accuracy compared to Cross-Encoders due to lack of interaction modeling between query and document.
  • Requires large embedding storage for all documents, which can be costly.
  • May struggle with nuanced semantic understanding without interaction modeling.
  • Less effective for short queries where context is crucial.
Ideal When
  • n > 1M documents
  • latency < 100ms
  • limited compute resources
  • batch processing of queries
Facebook uses Bi-Encoders for large-scale document retrieval where latency and scalability are critical.
Cross-Encoder
"High accuracy due to modeling interactions between query and document."
Weaknesses
  • 2-3× slower inference due to joint encoding of query and document.
  • Not scalable for large datasets due to need for pairwise comparison.
  • Higher computational cost, requiring more powerful hardware.
  • Increased latency, often exceeding 100ms for large datasets.
Ideal When
  • n < 100K documents
  • high accuracy requirement
  • sufficient compute resources
  • real-time query processing
Google uses Cross-Encoders for tasks requiring high precision, such as question answering systems.
🐍
Bi-Encoder - Example
▼ Show
Python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
query_embedding = model.encode('What is the capital of France?')
document_embeddings = model.encode(['Paris is the capital of France.', 'Berlin is the capital of Germany.'])
🐍
Cross-Encoder - Example
▼ Show
Python
from transformers import BertTokenizer, BertForSequenceClassification
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
inputs = tokenizer('What is the capital of France?', 'Paris is the capital of France.', return_tensors='pt')
outputs = model(**inputs)
✅
Use Bi-Encoder When
  • •n > 1M and latency < 100ms
  • •limited compute resources
  • •batch processing of queries
  • •embedding storage is manageable
✅
Use Cross-Encoder When
  • •n < 100K and high accuracy is critical
  • •sufficient compute resources available
  • •real-time query processing required
  • •interaction modeling is necessary
↳
Nuances
  • Bi-Encoders can be enhanced with reranking using Cross-Encoders for top results.
  • Cross-Encoders may not scale well but can be used for high-value queries.
  • Bi-Encoders are often used in conjunction with approximate nearest neighbor search to improve speed.
  • Cross-Encoders can be used for final reranking to balance speed and accuracy.

Practice

(1/5)
1. In an ML platform, which component is primarily responsible for ensuring that features are consistent across training and serving?
easy
A. Model Training Pipeline
B. Serving Infrastructure
C. Data Ingestion Pipeline
D. Feature Store

Solution

  1. Step 1: Identify the role of each component.

    The Feature Store is designed to ensure feature consistency across training and serving.

  2. Step 2: Eliminate wrong options.

    Model Training Pipeline focuses on training models, Serving Infrastructure handles model deployment, and Data Ingestion Pipeline deals with data collection.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feature Store ensures consistency. [OK]
Hint: Feature Store ensures feature consistency. [OK]
Common Mistakes:
2. For a real-time recommendation system, which metric should the ML platform optimize to ensure timely and relevant suggestions?
easy
A. Latency
B. Mean Absolute Error (MAE)
C. Precision
D. Recall

Solution

  1. Step 1: Understand the system's requirement.

    Real-time systems need to provide timely responses, making latency crucial.

  2. Step 2: Eliminate wrong options.

    MAE measures error, Precision and Recall focus on accuracy, not speed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time needs low latency. [OK]
Hint: Real-time systems prioritize low latency. [OK]
Common Mistakes:
3. In a news feed system designed for personalization at scale, which component is primarily responsible for filtering content based on user preferences?
easy
A. Content Delivery Network
B. Data Warehouse
C. Recommendation Engine
D. Load Balancer

Solution

  1. Step 1: Identify the role of each component

    The Recommendation Engine is responsible for filtering and suggesting content based on user preferences.

  2. Step 2: Eliminate wrong options

    A: Content Delivery Network is for content distribution, not personalization. C: Load Balancer manages traffic, not content filtering. D: Data Warehouse stores data, not directly involved in real-time personalization.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Recommendation Engine filters content for users. [OK]
Hint: Recommendation Engine filters content for users. [OK]
Common Mistakes:
4. In a spam detection system, which feature is most likely to be effective in identifying spam emails?
easy
A. Email length
B. Presence of certain keywords
C. Sender's email domain
D. Time of day email was sent

Solution

  1. Step 1: Identify effective features for spam detection

    Keywords like 'free', 'win', or 'prize' are often used in spam emails.

  2. Step 2: Eliminate less effective features

    Email length (A) and time of day (D) are not strong indicators of spam. Sender's domain (C) can be spoofed.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Keywords often signal spam. [OK]
Hint: Keywords often signal spam. [OK]
Common Mistakes:
5. In a delivery forecasting system, when should you prefer using a simple linear regression model over a complex deep learning model?
medium
A. When the dataset is extremely large and complex
B. When interpretability and simplicity are more important than accuracy
C. When the data has non-linear patterns
D. When the model needs to capture intricate temporal dependencies

Solution

  1. Step 1: Consider the trade-offs between model complexity and interpretability

    Simple models are easier to interpret and explain.

  2. Step 2: Eliminate wrong options

    Complex models are needed for large, complex datasets (A), non-linear patterns (C), and intricate dependencies (D).

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Choose simplicity for interpretability. [OK]
Hint: Choose simplicity for interpretability. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard