Bird
Raised Fist0

Two-Stage Retrieval - Candidate Generation Then Ranking

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Candidate Generation vs Ranking

In a large-scale recommendation system with a 100ms latency SLA, would you prioritize candidate generation or ranking for optimization? Justify your choice.

📊
Comparison
DimensionCandidate GenerationRanking
Efficiency in reducing dataset sizeHighMedium
Model complexityLowHigh
Inference latencyVery LowMedium
InterpretabilityHighLow
ScalabilityVery HighMedium
✅
Verdict
Default to candidate generation when dealing with very large datasets and strict latency requirements. Switch to ranking when you have a smaller candidate pool and can afford the computational cost for higher accuracy.
💡
Key Insight
Candidate generation efficiently narrows down options, while ranking provides precision - balance them based on dataset size and latency constraints.
⚠️
Interview Trap
❌ Wrong: Always prioritize ranking because it provides better accuracy.
This ignores the computational cost and latency constraints of ranking large datasets.
✅ Consider the trade-off between dataset size reduction and model complexity. Use candidate generation to efficiently narrow down options, then apply ranking for precision.
Candidate Generation
"Efficiently narrows down the search space to a manageable number of candidates."
Weaknesses
  • May miss relevant items if the candidate pool is too small.
  • Typically relies on simpler models, which might not capture complex user-item interactions.
  • Can introduce bias if the candidate generation model is not well-tuned.
  • Limited by the quality of the initial feature set used for generation.
Ideal When
  • When n > 1M and you need to reduce the dataset size quickly.
  • When latency < 50ms is critical for the first stage.
  • When you have limited computational resources for the initial filtering.
  • When the candidate pool needs to be refreshed frequently (e.g., every 5 minutes).
Spotify uses candidate generation to quickly filter millions of tracks to a few hundred before ranking.
Ranking
"Provides precise ordering of candidates based on complex models that capture nuanced interactions."
Weaknesses
  • Requires more computational resources, potentially increasing latency.
  • Complex models can be harder to interpret and debug.
  • Overfitting risk if the model is too complex for the available data.
  • Longer training times due to model complexity.
Ideal When
  • When n < 100K and you can afford detailed scoring.
  • When latency < 100ms is acceptable for the second stage.
  • When you have access to rich user-item interaction data.
  • When interpretability is less of a concern compared to accuracy.
Netflix uses sophisticated ranking models to order a small set of candidate movies for each user.
🐍
Candidate Generation - Example
▼ Show
Python
def generate_candidates(user_id, item_features, model):
    # Use a simple model like matrix factorization
    user_vector = model.get_user_vector(user_id)
    scores = item_features.dot(user_vector)
    top_candidates = np.argsort(scores)[-100:]
    return top_candidates
🐍
Ranking - Example
▼ Show
Python
def rank_candidates(user_id, candidates, model):
    # Use a complex model like a neural network
    user_features = model.get_user_features(user_id)
    candidate_scores = [model.predict(user_features, item) for item in candidates]
    ranked_candidates = sorted(zip(candidates, candidate_scores), key=lambda x: x[1], reverse=True)
    return [item for item, score in ranked_candidates]
✅
Use Candidate Generation When
  • •When n > 1M and you need to quickly reduce the dataset size.
  • •When you have a strict latency SLA < 50ms for the first stage.
  • •When computational resources are limited for initial filtering.
✅
Use Ranking When
  • •When n < 100K and detailed scoring is feasible.
  • •When you have rich interaction data and can afford higher latency.
  • •When model accuracy is prioritized over interpretability.
↳
Nuances
  • Candidate generation can be more effective when the initial dataset is extremely large, even if it sacrifices some accuracy.
  • Ranking models can overfit if the candidate pool is not diverse enough.
  • In systems with frequent updates, candidate generation needs to be highly efficient to keep up with data changes.
  • Complex ranking models may not justify their cost if the candidate generation stage already provides high-quality candidates.

Practice

(1/5)
1. For a delivery forecasting system, which metric is most appropriate to evaluate the accuracy of ETA predictions?
easy
A. Area Under the Curve (AUC)
B. F1 Score
C. Mean Absolute Error (MAE)
D. Precision

Solution

  1. Step 1: Identify the nature of the prediction task

    ETA prediction is a regression task, focusing on continuous values.

  2. Step 2: Eliminate wrong options

    F1 Score, AUC, and Precision are classification metrics. MAE is suitable for regression tasks.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    MAE for regression accuracy. [OK]
Hint: MAE for regression accuracy. [OK]
Common Mistakes:
2. For a hate speech detection system, which metric is most appropriate to optimize for minimizing false positives?
easy
A. Accuracy
B. Precision
C. Recall
D. F1-Score

Solution

  1. Step 1: Understand metric definitions

    Precision measures the ratio of true positives to the sum of true positives and false positives.

  2. Step 2: Eliminate wrong options

    Accuracy considers all classes, recall focuses on false negatives, and F1-score balances precision and recall.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Minimize false positives = Precision [OK]
Hint: Minimize false positives = Precision [OK]
Common Mistakes:
3. In a search ranking system, which component is primarily responsible for determining the relevance of a document to a query?
easy
A. Query parser
B. Ranking algorithm
C. Indexing module
D. Crawler

Solution

  1. Step 1: Identify the role of each component

    The ranking algorithm is responsible for determining the relevance of documents to a query.

  2. Step 2: Eliminate wrong options

    Indexing module organizes data, query parser interprets queries, and crawler collects data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Ranking algorithm determines relevance. [OK]
Hint: Ranking algorithm determines relevance. [OK]
Common Mistakes:
4. When implementing a Guardrails System for LLMs, should you prioritize precision over recall, and why?
medium
A. Yes, because precision ensures the system is efficient and fast.
B. No, because recall is more important to ensure all inappropriate content is caught.
C. Yes, to minimize false positives and avoid blocking legitimate content.
D. No, because precision and recall are equally important in all cases.

Solution

  1. Step 1: Understand the trade-off

    Prioritizing precision minimizes false positives, crucial for user experience.

  2. Step 2: Eliminate wrong options

    Recall is important but can lead to over-blocking, efficiency isn't the main concern, and equal importance isn't practical in all cases.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Prioritize precision to avoid blocking legitimate content. [OK]
Hint: Prioritize precision to avoid blocking legitimate content. [OK]
Common Mistakes:
5. When implementing a 'People You May Know' feature, should you use a simple Jaccard similarity or a more complex graph embedding approach? Assume the graph is sparse and dynamic.
medium
A. Use Jaccard similarity because it's computationally efficient for sparse graphs.
B. Use graph embeddings because they are always better than simple metrics.
C. Use Jaccard similarity because it adapts well to dynamic graphs.
D. Use graph embeddings because they capture more complex relationships.

Solution

  1. Step 1: Analyze graph characteristics.

    The graph is sparse and dynamic, requiring methods that capture complex relationships.

  2. Step 2: Evaluate options.

    Jaccard similarity (A) is efficient but too simplistic for capturing complex relationships. Graph embeddings (B) are suitable as they capture more nuanced relationships. Jaccard similarity (C) does not inherently adapt to dynamic graphs. Graph embeddings (D) are not always better, but in this case, they are due to the complexity of relationships.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Embeddings capture complex relationships. [OK]
Hint: Embeddings capture complex relationships. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard