Bird
Raised Fist0

Vector Search and Approximate Nearest Neighbor - FAISS, HNSW

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Vector Drift Dilemma
Started two weeks ago, ongoing.

A recommendation system using FAISS for vector search, serving 10M queries/day for a video streaming platform.

Symptom
Recommendation diversity collapsed, AUC dropped from 0.85 to 0.72 over the past two weeks.
📡
Signals
metric
Live AUC on recommendation system
Dropped from 0.85 to 0.72 over two weeks
distribution
Distribution of vector norms in the FAISS index
Shifted significantly, with a mean increase of 15%
latency
Real-time feature extraction latency
Spiked from P99 < 50ms to P99 = 120ms
log
Error logs from FAISS index updates
Increased frequency of 'index out of bounds' errors
feature_importance
Feature importance for 'user_engagement_score'
Dropped from 0.25 to 0.05 after last week's data refresh
🔬
Hypotheses
Vector norms have drifted due to a data preprocessing bug.
Supporting Signals
Distribution of vector norms in the FAISS indexError logs from FAISS index updates
Verify: Check preprocessing pipeline for changes in normalization logic.
df['vector_norm'] = np.linalg.norm(df['vectors'], axis=1); df['vector_norm'].plot()
Feature extraction latency is causing stale or incomplete data in the index.
Supporting Signals
Real-time feature extraction latencyError logs from FAISS index updates
Verify: Analyze feature extraction logs for bottlenecks or delays.
latency_df.plot(kind='line', x='timestamp', y='latency')
🎯
Root Cause
A bug in the data preprocessing pipeline caused incorrect normalization of vectors.
Why It Happened

A recent update to the data pipeline inadvertently altered the normalization logic, leading to vector drift.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the data preprocessing pipeline.
Short-termCorrect the normalization logic in the data preprocessing pipeline.
Long-termImplement a validation step in the pipeline to catch normalization errors before deployment.
🐍
Python Fix
Python
def normalize_vectors(vectors):
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    return vectors / norms

# Apply corrected normalization
data['vectors'] = normalize_vectors(data['vectors'])
✗
Common Wrong Diagnoses
▼ Show
The FAISS index is corrupted.
Why candidates think this: Frequent 'index out of bounds' errors suggest index corruption.
Why wrong: The errors were due to incorrect vector norms, not index corruption.
The model is overfitting to recent data.
Why candidates think this: AUC drop suggests overfitting.
Why wrong: The validation AUC remained stable, indicating the issue was not model overfitting.
📊
Monitoring to Add
  • Monitor vector norm distribution for significant shifts.
  • Set up alerts for spikes in feature extraction latency.
  • Track error rates in FAISS index updates.
📋
Process Change

Introduce a code review step specifically for data preprocessing changes.

🏗️
Architectural Change

Add a staging environment for testing data pipeline changes with a subset of live data.

💡
Key Insight
Vector norm drift due to preprocessing errors can silently degrade retrieval performance.

Practice

(1/5)
1. For an agent system designed to optimize task completion time, which metric should be prioritized?
easy
A. Latency
B. Accuracy
C. Precision
D. Recall

Solution

  1. Step 1: Understand the objective

    The goal is to optimize task completion time, which relates to how quickly tasks are executed.

  2. Step 2: Match metrics to objectives

    Latency measures the time delay in task execution, making it the relevant metric.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Task time = Latency focus [OK]
Hint: Task time optimization = focus on latency. [OK]
Common Mistakes:
2. For a recommendation system using collaborative filtering, which metric is most appropriate to evaluate the model's performance?
easy
A. Recall
B. Accuracy
C. Mean Squared Error
D. Precision

Solution

  1. Step 1: Identify the goal of recommendation systems.

    Recommendation systems aim to suggest relevant items, making precision a key metric.

  2. Step 2: Eliminate wrong options.

    Accuracy is too generic, MSE is for regression, and recall focuses on finding all relevant items, not precision.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Precision = relevant recommendations [OK]
Hint: Precision = relevant recommendations [OK]
Common Mistakes:
3. A bot detection system using device fingerprinting is failing to detect bots effectively. What is the most likely failure mode?
medium
A. The system is not considering user agent strings.
B. The system is not updating fingerprints frequently enough.
C. The system is overly reliant on IP addresses.
D. The system is using outdated behavioral models.

Solution

  1. Step 1: Identify the core issue with device fingerprinting.

    Device fingerprints need frequent updates to remain effective against evolving bots.

  2. Step 2: Eliminate wrong options.

    Behavioral models and IP reliance are not directly related to fingerprinting; user agent strings are not unique enough.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Frequent updates needed for effective fingerprinting [OK]
Hint: Frequent updates needed for effective fingerprinting [OK]
Common Mistakes:
4. In a search ranking system, when should you prefer a learning-to-rank model over a traditional BM25 algorithm?
medium
A. When you have a small dataset
B. When interpretability is crucial
C. When computational resources are limited
D. When you have rich user interaction data

Solution

  1. Step 1: Assess the data and resource context

    Learning-to-rank models excel with rich interaction data.

  2. Step 2: Evaluate the constraints

    BM25 is better for small datasets, interpretability, and limited resources.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use learning-to-rank with rich interaction data. [OK]
Hint: Use learning-to-rank with rich interaction data. [OK]
Common Mistakes:
5. As a recommendation system using neural collaborative filtering scales to 10x more users, which component is most likely to become a bottleneck?
hard
A. Model training time due to increased data volume.
B. Feature extraction pipeline.
C. Real-time recommendation latency.
D. Data storage for user-item interaction history.

Solution

  1. Step 1: Consider the impact of scaling on neural CF.

    Increased data volume significantly affects model training time.

  2. Step 2: Eliminate wrong options.

    Options A and D are less likely bottlenecks as storage and feature extraction can be optimized. Real-time recommendation latency. is affected but not as critically as training time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Scaling = training time bottleneck [OK]
Hint: Scaling = training time bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard