Bird
Raised Fist0

The Phantom Bot Surge

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Phantom Bot Surge
Started two weeks ago, ongoing

A bot detection system using behavioral signals and device fingerprinting for a large e-commerce platform, processing 5M requests per day.

Symptom
AUC dropped from 0.92 to 0.75 over the past two weeks · Increased false positives in bot detection, impacting legitimate users · Real-time feature extraction latency spiked from 50ms to 200ms
📡
Signals
metric
AUC on live traffic
Dropped from 0.92 to 0.75
latency
Real-time feature extraction latency
Spiked from 50ms to 200ms
distribution
Distribution of device fingerprint features
Shifted significantly post-deploy
feature_importance
Feature importance of 'device_type'
Dropped from 0.25 to 0.05
log
Error logs from feature extraction service
Increased frequency of 'timeout' errors
🔬
Hypotheses
Feature extraction service is causing latency spikes
Supporting Signals
latencylog
Verify: Check service logs for error rates and latency metrics
df_logs[df_logs['error'] == 'timeout'].groupby('timestamp').count().plot()
Device fingerprinting model is not adapting to new device types
Supporting Signals
distributionfeature_importance
Verify: Analyze recent device type distributions and feature importance changes
df_features.groupby('device_type').size().plot(kind='bar')
🎯
Root Cause
A recent deployment introduced a bug in the feature extraction service, causing timeouts and incorrect feature values.
Why It Happened

The deployment included untested changes to the feature extraction logic, leading to increased latency and incorrect feature values.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the feature extraction service
Short-termPatch the feature extraction service to handle new device types correctly
Long-termImplement a comprehensive testing framework for feature extraction changes
🐍
Python Fix
Python
def rollback_service():
    # Code to rollback service
    execute_rollback('feature_extraction_service')

rollback_service()
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to recent data
Why candidates think this: AUC drop suggests model performance issue
Why wrong: Validation AUC is stable, indicating the model itself is not overfitting
Network issues causing latency spikes
Why candidates think this: Latency spikes could be network-related
Why wrong: Latency spikes are specific to feature extraction service, not general network
📊
Monitoring to Add
  • Real-time latency monitoring for feature extraction service
  • Alert on significant shifts in feature distributions
  • Error rate monitoring for feature extraction service
📋
Process Change

Introduce mandatory regression testing for all feature extraction changes

🏗️
Architectural Change

Decouple feature extraction logic from deployment pipeline to allow independent testing and rollback

💡
Key Insight
Deployment changes in feature extraction can silently degrade model performance by altering feature distributions and latency.

Practice

(1/5)
1. In a production-scale image embedding pipeline, should you use a pre-trained model or train a model from scratch, given limited computational resources?
medium
A. Use a pre-trained model to save resources
B. Train from scratch for better customization
C. Always train from scratch for better accuracy
D. Use a pre-trained model only if data is scarce

Solution

  1. Step 1: Consider resource constraints.

    Pre-trained models are efficient and require less computational power.

  2. Step 2: Eliminate wrong options.

    Training from scratch is resource-intensive, and pre-trained models are not only for scarce data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Pre-trained saves resources. [OK]
Hint: Pre-trained saves resources. [OK]
Common Mistakes:
2. In an ML monitoring platform, when should you prioritize retraining the model over just alerting the team?
medium
A. When there's a significant drift detected in input data
B. When the model's accuracy drops slightly
C. When the model's predictions are consistent but incorrect
D. When the alerting system triggers multiple false positives

Solution

  1. Step 1: Identify scenarios for retraining

    Retraining is crucial when there's a significant drift in input data, indicating the model may no longer be valid.

  2. Step 2: Eliminate wrong options

    Minor accuracy drops or consistent incorrect predictions may not require retraining. False positives in alerts suggest tuning the alert system, not retraining.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Significant drift = Retrain [OK]
Hint: Significant drift = Retrain [OK]
Common Mistakes:
3. A RAG system is generating irrelevant responses. What is the most likely failure mode?
medium
A. The retriever is fetching irrelevant documents
B. The generator model is overfitting
C. The retriever is fetching too few documents
D. The encoder is not processing inputs correctly

Solution

  1. Step 1: Identify the source of irrelevance.

    If the retriever fetches irrelevant documents, the generator will produce irrelevant responses.

  2. Step 2: Eliminate wrong options.

    Fetching too few documents or overfitting would affect response quality, not relevance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Irrelevant input = irrelevant output. [OK]
Hint: Irrelevant input = irrelevant output. [OK]
Common Mistakes:
4. In a two-stage retrieval system, what is a likely failure mode if the candidate generation stage has a low recall?
medium
A. The ranking stage will have too many candidates to process.
B. The system will return irrelevant results to the user.
C. The ranking stage will not have enough relevant candidates to rank.
D. The system will have high computational costs in the ranking stage.

Solution

  1. Step 1: Understand the impact of low recall

    Low recall in candidate generation means not enough relevant items are passed to the ranking stage.

  2. Step 2: Eliminate wrong options

    The ranking stage will have too many candidates to process. is incorrect because low recall means fewer candidates, not more. The system will return irrelevant results to the user. is incorrect because irrelevant results are more about precision issues. The system will have high computational costs in the ranking stage. is incorrect because low recall reduces computational costs, not increases them.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Low recall means not enough relevant items. [OK]
Hint: Low recall means not enough relevant items. [OK]
Common Mistakes:
5. As the number of LLMs evaluated increases tenfold, which component of the evaluation system is most likely to fail first?
hard
A. Human evaluator availability
B. Benchmark task diversity
C. Data storage capacity
D. LLM-as-Judge processing power

Solution

  1. Step 1: Analyze scalability of components

    Human evaluators are a bottleneck as scaling requires more human resources.

  2. Step 2: Eliminate wrong options

    Data storage and processing power can be scaled with infrastructure, and task diversity is not a direct scalability issue.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Human evaluators are the bottleneck in scaling. [OK]
Hint: Human evaluators are the bottleneck in scaling. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard