Bird
Raised Fist0

The Relevance Recession

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Relevance Recession
The issue started two weeks ago and has been persistent since.

A search ranking system serving 10M queries per day, optimizing for query-document relevance.

Symptom
AUC dropped from 0.85 to 0.72 over the past two weeks, and user engagement metrics like click-through rate (CTR) have decreased by 15%.
📡
Signals
metric
AUC on live traffic
Dropped from 0.85 to 0.72 over two weeks
distribution
Query length distribution
Shifted towards longer queries in the past two weeks
feature_importance
Feature 'query_length' importance
Dropped from 0.15 to 0.05 after last model update
latency
Real-time feature extraction latency
Spiked from 50ms to 120ms P99
log
Error logs from feature extraction service
Increased frequency of 'timeout' errors
🔬
Hypotheses
Feature extraction latency is causing stale features to be used in ranking.
Supporting Signals
latencylog
Verify: Check if the feature extraction service is returning stale data due to timeouts.
stale_data_count = df[df['feature_timestamp'] < df['query_timestamp']]['feature_id'].count(); print(stale_data_count)
The model is not handling longer queries well due to a shift in query length distribution.
Supporting Signals
distributionfeature_importance
Verify: Analyze model performance on different query lengths.
df.groupby(pd.cut(df['query_length'], bins=[0, 5, 10, 15, 20]))['auc'].mean().plot(kind='bar')
🎯
Root Cause
The feature extraction service latency caused stale features to be used, degrading model performance.
Why It Happened

A recent deployment increased the complexity of feature extraction, leading to timeouts.

🔧
Fix Strategy
ImmediateRollback to the previous version of the feature extraction service.
Short-termOptimize the feature extraction code to reduce latency.
Long-termImplement a caching mechanism to ensure timely feature availability.
🐍
Python Fix
Python
def optimize_feature_extraction(data):
    # Simplified feature extraction logic
    return data.apply(lambda x: x * 0.9)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The drop in AUC suggests a generalization issue.
Why wrong: The validation AUC remains stable, indicating the model generalizes well on the validation set.
A data pipeline issue caused incorrect labels.
Why candidates think this: Label issues often cause performance drops.
Why wrong: Label distribution checks show no anomalies.
📊
Monitoring to Add
  • Real-time feature extraction latency alerts
  • AUC monitoring on different query lengths
  • Error rate monitoring for feature extraction service
📋
Process Change

Implement a rollback plan for feature extraction service changes.

🏗️
Architectural Change

Introduce a feature extraction service with built-in caching and redundancy.

💡
Key Insight
Feature extraction latency can silently degrade model performance by causing stale data usage.

Practice

(1/5)
1. When implementing a Guardrails System for LLMs, should you prioritize precision over recall, and why?
medium
A. Yes, because precision ensures the system is efficient and fast.
B. No, because recall is more important to ensure all inappropriate content is caught.
C. Yes, to minimize false positives and avoid blocking legitimate content.
D. No, because precision and recall are equally important in all cases.

Solution

  1. Step 1: Understand the trade-off

    Prioritizing precision minimizes false positives, crucial for user experience.

  2. Step 2: Eliminate wrong options

    Recall is important but can lead to over-blocking, efficiency isn't the main concern, and equal importance isn't practical in all cases.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Prioritize precision to avoid blocking legitimate content. [OK]
Hint: Prioritize precision to avoid blocking legitimate content. [OK]
Common Mistakes:
2. As the number of users and ads increases tenfold in an ad auction system, which component is most likely to fail first?
hard
A. User Data Storage
B. Ad Ranking Module
C. Bid Management System
D. Network Bandwidth

Solution

  1. Step 1: Consider scale impact

    With a tenfold increase in users and ads, the User Data Storage is likely to face capacity issues first due to the sheer volume of data.

  2. Step 2: Eliminate wrong options

    • Ad Ranking Module: Computationally intensive but not storage-bound.
    • Bid Management System: Handles bids, not directly affected by user volume.
    • Network Bandwidth: Affects data transfer, but storage is a more immediate bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Storage fails first with data volume increase. [OK]
Hint: Storage fails first with data volume increase. [OK]
Common Mistakes:
3. As the user base scales 10x, which component of a DLRM-based CTR prediction system is most likely to become a bottleneck?
hard
A. Data preprocessing pipeline
B. Training data storage capacity
C. Model inference latency
D. Embedding table size

Solution

  1. Step 1: Consider scaling effects on components.

    Embedding tables grow with user and feature diversity.

  2. Step 2: Eliminate wrong options.

    Preprocessing (A) and inference (B) scale with compute resources. Storage (D) is less critical than real-time constraints.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Embedding tables grow with scale. [OK]
Hint: Embedding tables grow with scale. [OK]
Common Mistakes:
4. As the number of users increases tenfold in a delivery forecasting system, which component is most likely to become a bottleneck first?
hard
A. Real-time prediction latency
B. Model training time
C. Feature engineering pipeline
D. Data storage capacity

Solution

  1. Step 1: Consider the impact of user scale on system components

    Real-time prediction latency is critical as user requests increase.

  2. Step 2: Eliminate wrong options

    Data storage (A) and feature engineering (D) scale more linearly. Model training (B) is periodic, not real-time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Latency is the first bottleneck at scale. [OK]
Hint: Latency is the first bottleneck at scale. [OK]
Common Mistakes:
5. As the dataset size increases 10x in a FAISS-based vector search system, which component is most likely to become a bottleneck first?
hard
A. Index storage
B. Search algorithm
C. Dimensionality reduction
D. Quantization process

Solution

  1. Step 1: Consider the impact of dataset size increase.

    Index storage will grow significantly, potentially becoming a bottleneck.

  2. Step 2: Eliminate wrong options.

    Search algorithm, dimensionality reduction, and quantization are less affected by size increase compared to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Index storage grows with dataset size. [OK]
Hint: Index storage grows with dataset size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard