Bird
Raised Fist0

RAG - Retrieval-Augmented Generation System Design

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
RAG - Retrieval-Augmented Generation System Design
Design a system that combines retrieval with generation for enhanced response quality.
system-designhardFreq: very_highGenAI Systems
Prerequisites:Semantic SearchVector Search and ANNLLM Serving
🎯
Problem Framing
Business Objective

Enhance user engagement by providing accurate and contextually relevant responses in a conversational AI system.

ML Objective

Maximize the relevance and accuracy of generated responses by combining retrieval with generation.

Constraints
  • P99 latency < 100ms for response generation
  • High throughput of 1000 QPS
  • Limited labeled data for training
  • Compliance with data privacy regulations
  • Handling of diverse and multilingual queries
Clarifying Questions
  • What is the expected scale of user queries per day?
  • Are there specific domains or topics that the system should prioritize?
  • What are the latency requirements for the system?
  • How often does the underlying knowledge base update?
  • What fallback mechanisms are acceptable if the model fails?
Out of Scope
✗ Building the underlying knowledge base
✗ Developing a new language model from scratch
✗ Handling user authentication and authorization
📊
Offline Metrics
Recall@K: Measures the proportion of relevant documents retrieved, crucial for ensuring the retrieval component captures necessary context.
BLEU Score: Evaluates the quality of generated text against reference responses, important for assessing generation quality.
📈
Online Metrics
CTR: Measures user engagement with the generated responses.
User Satisfaction Score: Direct feedback from users on the relevance and helpfulness of responses.
⚖️
Trade-offs

High recall in retrieval might lead to lower precision, affecting BLEU score. Balancing retrieval breadth with generation quality is key.

💼
Business Metric

Overall user engagement and satisfaction with the conversational AI system.

🗄️
Data Sources
  • User query logs (text, real-time)
  • Knowledge base documents (text, updated weekly)
  • User feedback and ratings (numerical, real-time)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions and explicit ratings for response quality.

Data Volume

Approximately 10 million queries per day, 500GB of text data per month.

Freshness

Features should be updated in real-time to reflect the latest user interactions and knowledge base updates.

⚠️
Data Challenges
  • Sparse labels due to limited explicit feedback
  • Position bias in user interactions
  • Cold start for new topics or domains
  • Handling multilingual data effectively
🔧
Key Features
embedding
Query Embeddings
Captures semantic meaning of user queries for effective retrieval.
embedding
Document Embeddings
Represents knowledge base documents for similarity matching.
categorical
User Interaction History
Personalizes responses based on past interactions.
numerical
Response Quality Score
Provides feedback loop for improving generation quality.
categorical
Language Detection
Ensures responses are generated in the correct language.
⚡
Feature Engineering Notes
Real-time vs Batch

Query embeddings and language detection need real-time computation, while document embeddings can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_query_embedding(query, model):
    return model.encode(query)

query_embedding = compute_query_embedding(user_query, embedding_model)
🤖
Model Selection
Baseline

TF-IDF based retrieval with a simple rule-based generation.

Proposed Model

Two-tower model for retrieval combined with a fine-tuned transformer for generation.

The two-tower model allows efficient retrieval by precomputing document embeddings, while the transformer ensures high-quality generation.
✗
Alternatives Rejected
XGBoost: Rejected due to inability to handle high-dimensional embeddings effectively.
Single-stage transformer: Rejected due to high latency in generating responses without retrieval augmentation.
🏋️
Training Strategy

Hybrid approach with offline training for retrieval and online fine-tuning for generation based on user feedback.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput1000 QPS
ArchitectureTwo-stage architecture with retrieval followed by generation to balance latency and response quality.
CachingCache popular query results with a TTL of 5 minutes to reduce load and latency.
FallbackUse rule-based responses or cached results if the model is unavailable.
📡
Data Drift Signals
Shift in query embedding distribution
Change in language distribution of queries
Decrease in retrieval recall
📉
Model Drift Signals
Drop in BLEU score
Increase in user dissatisfaction reports
Shift in prediction score distribution
🔔
Alerting & Retraining
Retraining Trigger

Retrain when BLEU score drops by more than 10% over a month.

Alerting Strategy

Page on-call engineer if latency exceeds 150ms or if user dissatisfaction reports increase by 20%.

🎯
Interview Angles
  • How do you balance retrieval quality with generation latency?
  • What strategies do you use to handle data sparsity in feedback?
  • How do you ensure compliance with data privacy regulations?
⚠️
Common Mistakes
Ignoring the latency impact of complex generation models.
Overfitting to offline metrics without considering online user feedback.
Neglecting multilingual support in feature engineering.
📌
Key Decisions
  • Choosing a two-stage architecture to balance retrieval and generation.
  • Implementing a hybrid training strategy to leverage both offline and online data.
  • Designing a caching strategy to meet latency requirements.

Practice

(1/5)
1. In the context of agent system design, which component is primarily responsible for deciding the sequence of actions to achieve a goal?
easy
A. Memory Architecture
B. Planning
C. Tool Use
D. Data Preprocessing

Solution

  1. Step 1: Identify the role of each component

    Planning is responsible for deciding the sequence of actions to achieve a goal.

  2. Step 2: Eliminate wrong options

    Memory Architecture is for storing information, Tool Use is for executing actions, and Data Preprocessing is unrelated to action sequencing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Planning = action sequence decision [OK]
Hint: Planning decides action sequences. [OK]
Common Mistakes:
2. For a Guardrails System in LLMs, which metric is most appropriate to evaluate the effectiveness of content moderation?
easy
A. F1 Score
B. Precision
C. Recall
D. Accuracy

Solution

  1. Step 1: Understand the importance of balance

    F1 Score balances precision and recall, crucial for content moderation.

  2. Step 2: Eliminate wrong options

    Accuracy doesn't account for class imbalance, precision alone ignores false negatives, and recall alone ignores false positives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    F1 Score balances precision and recall. [OK]
Hint: F1 Score balances precision and recall. [OK]
Common Mistakes:
3. A delivery forecasting model is consistently overestimating ETAs during peak traffic hours. What is the most likely failure mode?
medium
A. The model is not accounting for real-time traffic data
B. The model has too many parameters
C. The model is underfitting the data
D. The model is overfitting to the training data

Solution

  1. Step 1: Identify the cause of overestimation during peak hours

    Real-time traffic data is crucial for accurate ETA predictions.

  2. Step 2: Eliminate wrong options

    Underfitting (A) and overfitting (D) are general issues, not specific to peak hours. Too many parameters (C) would cause overfitting, not specific time errors.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time data is key for peak accuracy. [OK]
Hint: Real-time data is key for peak accuracy. [OK]
Common Mistakes:
4. In a feed ranking system, should you prioritize using collaborative filtering or content-based filtering when user data is sparse?
medium
A. Collaborative filtering, because it leverages user interactions
B. Content-based filtering, because it relies on item features
C. Collaborative filtering, because it scales better with more users
D. Content-based filtering, because it requires less computational power

Solution

  1. Step 1: Analyze data sparsity

    When user data is sparse, collaborative filtering struggles due to lack of interaction data.

  2. Step 2: Eliminate wrong options

    Collaborative filtering needs user interactions. Content-based filtering uses item features, making it suitable for sparse data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Content-based works with sparse data. [OK]
Hint: Content-based works with sparse data. [OK]
Common Mistakes:
5. In an ML monitoring platform, when should you prioritize retraining the model over just alerting the team?
medium
A. When there's a significant drift detected in input data
B. When the model's accuracy drops slightly
C. When the model's predictions are consistent but incorrect
D. When the alerting system triggers multiple false positives

Solution

  1. Step 1: Identify scenarios for retraining

    Retraining is crucial when there's a significant drift in input data, indicating the model may no longer be valid.

  2. Step 2: Eliminate wrong options

    Minor accuracy drops or consistent incorrect predictions may not require retraining. False positives in alerts suggest tuning the alert system, not retraining.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Significant drift = Retrain [OK]
Hint: Significant drift = Retrain [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard