Bird
Raised Fist0

The Retrieval Collapse

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Retrieval Collapse
The issue started three days ago and has been ongoing since.

A Retrieval-Augmented Generation (RAG) system for a customer support chatbot serving 500k queries per day.

Symptom
The system's AUC dropped from 0.85 to 0.68 on live traffic, and recommendation diversity collapsed, with repeated retrievals of the same documents.
📡
Signals
metric
Live AUC observed in production environment
AUC dropped from 0.85 to 0.68
distribution
Distribution of retrieved document IDs
80% of queries retrieve the same document ID repeatedly
latency
Real-time feature extraction latency
Latency spiked from 50ms to 200ms P99
feature_importance
Feature importance for 'query_contextual_embedding'
Importance dropped from 0.25 to 0.05 after last update
log
Error logs from retrieval component
Increased frequency of 'timeout' errors in retrieval logs
🔬
Hypotheses
The retrieval model is overfitting to a small subset of documents.
Supporting Signals
Distribution of retrieved document IDsFeature importance for 'query_contextual_embedding'
Verify: Check the distribution of document retrievals over the past week.
import pandas as pd
retrievals = pd.read_csv('retrieval_logs.csv')
retrievals['document_id'].value_counts().head(10).plot(kind='bar')
Feature extraction latency is causing timeouts in retrieval, leading to fallback behavior.
Supporting Signals
Real-time feature extraction latencyError logs from retrieval component
Verify: Analyze latency logs and correlate with timeout errors.
latency_logs = pd.read_csv('latency_logs.csv')
timeout_logs = pd.read_csv('timeout_logs.csv')
merged_logs = pd.merge(latency_logs, timeout_logs, on='timestamp')
merged_logs.plot(x='timestamp', y=['latency', 'timeouts'])
🎯
Root Cause
The retrieval component is timing out due to increased feature extraction latency, causing fallback to a default document.
Why It Happened

A recent deployment introduced a bug in the feature extraction pipeline, increasing latency and causing retrieval timeouts.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the feature extraction pipeline.
Short-termIdentify and fix the bug in the feature extraction code causing increased latency.
Long-termImplement a robust timeout handling mechanism in the retrieval component to prevent fallback to default documents.
🐍
Python Fix
Python
# Example fix for feature extraction latency
import time

def extract_features(query):
    start_time = time.time()
    features = complex_feature_extraction(query)
    if time.time() - start_time > 0.1:  # 100ms threshold
        raise TimeoutError('Feature extraction took too long')
    return features
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to recent data.
Why candidates think this: The drop in AUC suggests a model issue.
Why wrong: Validation AUC is stable, indicating the model itself is not overfitting.
Data distribution shift in live traffic.
Why candidates think this: AUC drop could indicate a shift.
Why wrong: The issue is with retrieval timeouts, not data distribution.
📊
Monitoring to Add
  • Real-time feature extraction latency monitoring with alerts for spikes
  • Distribution monitoring of retrieved document IDs
  • Timeout error rate monitoring in retrieval component
  • AUC monitoring on live traffic with alert thresholds
📋
Process Change

Implement a canary deployment process for feature extraction changes to catch latency issues early.

🏗️
Architectural Change

Decouple feature extraction from retrieval with an asynchronous processing queue to handle spikes without timeouts.

💡
Key Insight
Retrieval timeouts due to latency spikes can cause fallback behavior, masking as model degradation.

Practice

(1/5)
1. In the context of hate speech detection, which component is responsible for transforming raw text data into numerical features?
easy
A. Feature Extraction
B. Data Augmentation
C. Model Training
D. Hyperparameter Tuning

Solution

  1. Step 1: Identify the role of each component

    Feature extraction is the process of converting raw text into numerical features that can be used by machine learning models.

  2. Step 2: Eliminate wrong options

    Data augmentation is about creating more data, model training is about learning from features, and hyperparameter tuning is about optimizing model parameters.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Features from text = Feature Extraction [OK]
Hint: Features from text = Feature Extraction [OK]
Common Mistakes:
2. In a multimodal system that integrates vision and language models, which component is responsible for aligning the semantic meaning between image features and text embeddings?
easy
A. Cross-modal attention layer
B. Text encoder
C. Image encoder
D. Output classifier

Solution

  1. Step 1: Identify the role of each component

    The cross-modal attention layer is designed to align and integrate features from different modalities, such as images and text.

  2. Step 2: Eliminate wrong options

    Image encoder and text encoder process their respective modalities independently. The output classifier is responsible for final predictions, not alignment.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cross-modal aligns semantics between modalities. [OK]
Hint: Cross-modal aligns semantics between modalities. [OK]
Common Mistakes:
3. As the number of concurrent users increases tenfold, which component of the LLM serving infrastructure is most likely to become a bottleneck first?
hard
A. Network bandwidth
B. Batching mechanism
C. Speculative Decoding
D. KV Cache

Solution

  1. Step 1: Consider the impact of scaling.

    Network bandwidth is often the first bottleneck as user requests increase.

  2. Step 2: Eliminate wrong options.

    Batching can handle more requests efficiently. KV Cache scales with memory, not users. Speculative Decoding affects processing, not network.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Network bandwidth limits scale first. [OK]
Hint: Network bandwidth limits scale first. [OK]
Common Mistakes:
4. As the data volume increases tenfold in an ML monitoring platform, which component is most likely to become a bottleneck first?
hard
A. Retraining Scheduler
B. Alerting System
C. Drift Detection Module
D. Model Evaluation Component

Solution

  1. Step 1: Analyze component scalability

    The Drift Detection Module processes incoming data to identify changes, making it sensitive to data volume increases.

  2. Step 2: Consider computational load

    Drift detection often involves statistical tests or complex calculations, which can become a bottleneck with large data volumes.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Data increase = Drift bottleneck [OK]
Hint: Data increase = Drift bottleneck [OK]
Common Mistakes:
5. As the number of users and items grows significantly, which component of a Two-Tower Model for Retrieval is likely to become a bottleneck first?
hard
A. Similarity score computation
B. Item embedding computation
C. User embedding computation
D. Model training time

Solution

  1. Step 1: Consider the scaling impact on components

    As users and items grow, computing similarity scores between all pairs becomes computationally expensive.

  2. Step 2: Eliminate wrong options

    User embedding computation and B scale linearly with users and items. Model training time is affected but not as immediately as similarity computation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Similarity computation scales poorly with size. [OK]
Hint: Similarity computation scales poorly with size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard