Bird
Raised Fist0

The Misjudged Model

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Misjudged Model
The issue started two weeks ago and has been persistent since.

An LLM evaluation system that uses benchmarks, human evaluations, and LLM-as-Judge to assess model performance. The system processes 500K evaluations per day.

Symptom
The system's AUC for evaluating model outputs dropped from 0.85 to 0.72 over the past two weeks.
📡
Signals
metric
AUC on live evaluations dropped significantly.
AUC dropped from 0.85 to 0.72
distribution
Distribution of evaluation scores from LLM-as-Judge shows a shift.
Mean score shifted from 0.75 to 0.60
feature_importance
Feature importance for 'contextual_relevance' dropped.
Importance dropped from 0.22 to 0.05
latency
Latency for LLM-as-Judge evaluations increased.
P99 latency increased from 150ms to 300ms
log
Error logs show increased rate of 'context not found' errors.
Error rate increased by 30%
🔬
Hypotheses
The LLM-as-Judge model is misconfigured or using outdated weights.
Supporting Signals
AUC dropDistribution shiftFeature importance drop
Verify: Check the model configuration and weight versioning logs.
print(model.config); print(model.version)
Data pipeline issues causing context misalignment in evaluations.
Supporting Signals
Error logsLatency increase
Verify: Inspect data pipeline logs for recent changes or errors.
df_logs[df_logs['error'] == 'context not found'].tail()
🎯
Root Cause
The LLM-as-Judge model was inadvertently rolled back to an older version during a deployment.
Why It Happened

A deployment script error caused the rollback to an older model version.

🔧
Fix Strategy
ImmediateRollback to the last known good model version.
Short-termCorrect the deployment script to prevent version rollbacks.
Long-termImplement version control checks and automated alerts for model version mismatches.
🐍
Python Fix
Python
model.load_weights('latest_good_version'); deploy_script.fix_versioning()
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The AUC drop suggests a performance issue.
Why wrong: The validation AUC remains stable, indicating the model itself is not overfitting.
The data pipeline is introducing noise.
Why candidates think this: The error logs and latency increase suggest data issues.
Why wrong: The root cause was a model version rollback, not data noise.
📊
Monitoring to Add
  • Alert on AUC drops below 0.80
  • Monitor model version changes in real-time
  • Track distribution shifts in evaluation scores
📋
Process Change

Implement a deployment checklist that includes version verification.

🏗️
Architectural Change

Add a version control system that locks model versions post-deployment.

💡
Key Insight
Always verify model versioning during deployments to prevent accidental rollbacks.

Practice

(1/5)
1. For a feed ranking system, which metric should be optimized to ensure users see the most relevant content?
easy
A. Click-through rate (CTR)
B. Page load time
C. Number of ads displayed
D. Total time spent on platform

Solution

  1. Step 1: Understand the goal of relevance

    CTR measures how often users click on posts, indicating relevance.

  2. Step 2: Eliminate wrong options

    Page load time affects performance, not relevance. Number of ads and total time spent are not direct indicators of content relevance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CTR indicates relevance. [OK]
Hint: CTR indicates relevance. [OK]
Common Mistakes:
2. For a semantic search system using a Bi-Encoder, which metric is most appropriate to evaluate the retrieval performance?
easy
A. Accuracy
B. Precision@K
C. Mean Squared Error
D. F1 Score

Solution

  1. Step 1: Understand retrieval evaluation metrics

    Precision@K measures the relevance of the top K retrieved documents, which is crucial for retrieval tasks.

  2. Step 2: Eliminate wrong options

    Accuracy and F1 Score are more suited for classification tasks, Mean Squared Error is for regression.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Precision@K = retrieval relevance [check]
Hint: Precision@K = retrieval relevance [check]
Common Mistakes:
3. In an HNSW-based vector search system, what is a likely failure mode if the graph's maximum layer is not properly maintained?
medium
A. Increased precision but decreased recall
B. Increased memory usage without affecting speed
C. Decreased search speed due to inefficient traversal
D. Decreased indexing speed but improved search accuracy

Solution

  1. Step 1: Understand the role of the maximum layer.

    The maximum layer affects traversal efficiency.

  2. Step 2: Eliminate wrong options.

    Precision, recall, and memory usage are not directly affected by the maximum layer.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Max layer affects traversal speed. [OK]
Hint: Max layer affects traversal speed. [OK]
Common Mistakes:
4. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
5. As the user base of an ML platform grows 10x, which component is most likely to become a bottleneck first?
hard
A. Model Training Pipeline
B. Feature Store
C. Serving Infrastructure
D. Data Ingestion Pipeline

Solution

  1. Step 1: Consider the impact of scaling on each component.

    Serving Infrastructure is directly impacted by increased user requests, making it a likely bottleneck.

  2. Step 2: Eliminate wrong options.

    Model Training and Data Ingestion are less affected by user growth, and Feature Store scales with data, not users.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Serving scales with user requests. [OK]
Hint: Serving scales with user requests. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard