Bird
Raised Fist0

Two-Stage Retrieval - Candidate Generation Then Ranking

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Design a Two-Stage Retrieval System for a Large-Scale E-commerce Platform
45 min · senior
senior

You are tasked with designing a two-stage retrieval system for a large e-commerce platform with 50 million DAU. The system must generate personalized product recommendations. You have 6 months and a team of 5 engineers. Implicit feedback data (clicks, purchases) is available, but explicit ratings are not. The P99 latency SLA for the candidate generation stage is 100ms, and for the ranking stage is 50ms.

Given Info
  • 50 million DAU
  • Implicit feedback data available
  • No explicit ratings
  • P99 latency SLA: 100ms for candidate generation, 50ms for ranking
  • Team of 5 engineers, 6-month timeline
📋
Guided Steps
1/5
Step 1
Candidate Generation Strategy
How will you design the candidate generation stage to handle 50 million DAU while meeting the 100ms latency SLA?
💡 Hint
Consider the trade-off between model complexity and inference speed.
Key Decision
Use approximate nearest neighbor (ANN) search because it balances speed and scalability for large datasets.
Interviewer Expectation
Mid: Proposes a basic ANN approach. Senior: Justifies the choice with specific latency and scalability benefits, and considers data partitioning strategies.
⚠️Common Wrong Answer
Using a complex deep learning model for candidate generation, leading to unacceptable latency.
Step 2
Ranking Model Selection
What model will you use for the ranking stage, and how will it meet the 50ms latency requirement?
💡 Hint
Think about models that can leverage implicit feedback effectively.
Key Decision
Choose a gradient boosting machine (GBM) because it efficiently handles implicit feedback and provides fast inference.
Interviewer Expectation
Mid: Selects GBM for its speed. Senior: Also discusses feature engineering to optimize latency and accuracy.
⚠️Common Wrong Answer
Selecting a deep neural network without considering its inference time.
Step 3
Data Pipeline Design
How will you design the data pipeline to ensure data freshness and handle implicit feedback?
💡 Hint
Consider the impact of data latency on model performance.
Key Decision
Implement a near-real-time data pipeline using stream processing to ensure data freshness.
Interviewer Expectation
Mid: Proposes a basic batch processing pipeline. Senior: Incorporates stream processing for real-time updates and discusses handling data skew.
⚠️Common Wrong Answer
Relying solely on batch processing, leading to stale data issues.
Step 4
Evaluation Metrics
What metrics will you use to evaluate the system both offline and online?
💡 Hint
Differentiate between metrics that measure model performance and those that measure business impact.
Key Decision
Use PR-AUC for offline evaluation and CTR plus revenue impact for online evaluation.
Interviewer Expectation
Mid: Mentions basic metrics like accuracy. Senior: Explains the choice of PR-AUC for imbalanced data and connects CTR to business goals.
⚠️Common Wrong Answer
Focusing only on offline metrics like accuracy, ignoring business impact.
Step 5
Scalability and Maintenance
How will you ensure the system scales with user growth and remains maintainable?
💡 Hint
Think about modularity and the ability to update models without downtime.
Key Decision
Design a modular architecture with separate components for candidate generation and ranking to allow independent scaling and updates.
Interviewer Expectation
Mid: Proposes a basic scalable architecture. Senior: Discusses modularity, A/B testing frameworks, and CI/CD pipelines for model updates.
⚠️Common Wrong Answer
Creating a monolithic system that is difficult to scale and maintain.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Uses ANN for candidate generation to meet latency requirements.
  • Selects GBM for ranking due to its speed and effectiveness with implicit feedback.
  • Designs a data pipeline that ensures data freshness with stream processing.
  • Distinguishes between offline (PR-AUC) and online (CTR, revenue) metrics.
  • Proposes a modular architecture for scalability and maintainability.
Good to Have - 30%
  • Considers data partitioning strategies for ANN scalability.
  • Discusses feature engineering to optimize GBM performance.
  • Incorporates real-time updates in the data pipeline.
  • Connects CTR to business goals in metric selection.
  • Includes A/B testing frameworks in the architecture.
Bonus - 10%
  • Quantifies latency benefits of ANN over other methods.
  • Explains handling of data skew in stream processing.
  • Proposes CI/CD pipelines for seamless model updates.
✅
Model Answer
▼ Reveal

The ideal design uses an approximate nearest neighbor (ANN) search for candidate generation to handle large-scale data efficiently within the 100ms latency SLA. For ranking, a gradient boosting machine (GBM) is chosen for its fast inference and ability to leverage implicit feedback. The data pipeline is designed with stream processing to ensure data freshness. Evaluation metrics include PR-AUC for offline and CTR plus revenue impact for online performance. The architecture is modular to support scalability and maintainability.

Key Decisions
  • ANN for candidate generation due to its speed and scalability.
  • GBM for ranking because it handles implicit feedback well and meets latency requirements.
  • Stream processing in the data pipeline to maintain data freshness.
  • PR-AUC for offline evaluation and CTR plus revenue impact for online evaluation.
  • Modular architecture to allow independent scaling and updates.
Anti-patterns
Using complex deep learning models for candidate generation, leading to high latency.
Ignoring data freshness by relying solely on batch processing.
Focusing only on offline metrics without considering business impact.
Creating a monolithic system that is hard to scale and maintain.

Practice

(1/5)
1. In an ad auction system, which component is responsible for estimating the expected value of an ad based on historical click-through rates and bid amounts?
easy
A. User Targeting Engine
B. Bid Management System
C. Ad Ranking Module
D. Ad Inventory Manager

Solution

  1. Step 1: Identify the component's role

    The Ad Ranking Module is responsible for estimating the expected value of an ad by considering factors like historical click-through rates and bid amounts.

  2. Step 2: Eliminate wrong options

    • Bid Management System: Focuses on managing bids, not estimating ad value.
    • User Targeting Engine: Targets users, not involved in value estimation.
    • Ad Inventory Manager: Manages ad inventory, not value estimation.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Ad Ranking estimates value. [OK]
Hint: Ad Ranking Module estimates ad value. [OK]
Common Mistakes:
2. In the context of recommendation systems, which of the following best describes collaborative filtering?
easy
A. Using user-item interaction data to recommend items.
B. Using user demographic data to recommend items.
C. Using item metadata to recommend similar items.
D. Using content-based features to recommend items.

Solution

  1. Step 1: Understand collaborative filtering.

    Collaborative filtering relies on user-item interaction data to make recommendations.

  2. Step 2: Eliminate wrong options.

    Options A, B, and D describe content-based or demographic-based approaches, not collaborative filtering.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Collaborative = user-item interactions [OK]
Hint: Collaborative = user-item interactions [OK]
Common Mistakes:
3. When evaluating LLMs, why might you choose human evaluation over LLM-as-Judge?
medium
A. Human evaluation is faster and more scalable.
B. LLM-as-Judge provides more consistent results.
C. Human evaluation is less subjective.
D. LLM-as-Judge is biased towards its training data.

Solution

  1. Step 1: Compare human evaluation and LLM-as-Judge

    LLM-as-Judge can be biased by its training data, while human evaluation can provide diverse perspectives.

  2. Step 2: Eliminate wrong options

    Human evaluation is not faster or more scalable, and it is more subjective than LLM-as-Judge.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    LLM-as-Judge bias vs. human diversity. [OK]
Hint: LLM-as-Judge bias vs. human diversity. [OK]
Common Mistakes:
4. In an ML monitoring platform, when should you prioritize retraining the model over just alerting the team?
medium
A. When there's a significant drift detected in input data
B. When the model's accuracy drops slightly
C. When the model's predictions are consistent but incorrect
D. When the alerting system triggers multiple false positives

Solution

  1. Step 1: Identify scenarios for retraining

    Retraining is crucial when there's a significant drift in input data, indicating the model may no longer be valid.

  2. Step 2: Eliminate wrong options

    Minor accuracy drops or consistent incorrect predictions may not require retraining. False positives in alerts suggest tuning the alert system, not retraining.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Significant drift = Retrain [OK]
Hint: Significant drift = Retrain [OK]
Common Mistakes:
5. As the user base of a Guardrails System for LLMs scales 10x, which component is most likely to become a bottleneck first?
hard
A. Model training module
B. User feedback loop
C. Data storage system
D. Content filtering module

Solution

  1. Step 1: Analyze component scalability

    Content filtering must process each request, becoming a bottleneck as requests increase.

  2. Step 2: Eliminate wrong options

    Feedback loops and model training are less frequent, and data storage scales more linearly.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Content filtering scales with user requests. [OK]
Hint: Content filtering scales with user requests. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard