Bird
Raised Fist0

The Intent Misunderstanding

SYSTEM-DESIGNmediumInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Intent Misunderstanding
Started last Friday, ongoing for 3 days.

Intent classification and entity extraction system for a voice assistant serving 5M queries/day.

Symptom
Model achieves 0.92 AUC offline but 0.75 AUC on live traffic. Entity extraction accuracy dropped from 85% to 60%.
📡
Signals
metric
Live AUC for intent classification
Dropped from 0.92 to 0.75
metric
Entity extraction accuracy
Dropped from 85% to 60%
distribution
Distribution of query lengths
Shifted towards shorter queries post-deploy
feature_importance
Feature importance for 'query_length'
Dropped from 0.15 to 0.03 after last Friday's deploy
latency
Real-time feature extraction latency
Spiked from 50ms to 150ms P99
🔬
Hypotheses
Feature extraction latency causing timeouts
Supporting Signals
Real-time feature extraction latency spiked
Verify: Check server logs for timeout errors
import pandas as pd; df = pd.read_csv('server_logs.csv'); df[df['error'] == 'timeout'].count()
Query length distribution shift affecting model performance
Supporting Signals
Distribution of query lengths shiftedFeature importance for 'query_length' dropped
Verify: Analyze query length distribution pre and post-deploy
import matplotlib.pyplot as plt; df = pd.read_csv('queries.csv'); df['query_length'].hist(by=df['date'] > '2023-10-01'); plt.show()
🎯
Root Cause
Query length distribution shift due to a change in user behavior post-deploy.
Why It Happened

A new feature encouraged users to input shorter queries, which the model wasn't trained on.

🔧
Fix Strategy
ImmediateRollback to the previous model version.
Short-termUpdate the model with recent data reflecting the new query length distribution.
Long-termImplement a dynamic retraining pipeline that adapts to distribution shifts.
🐍
Python Fix
Python
import pandas as pd; from sklearn.model_selection import train_test_split; df = pd.read_csv('recent_queries.csv'); X_train, X_test, y_train, y_test = train_test_split(df['query'], df['intent'], test_size=0.2, random_state=42); # Retrain model with updated data
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to small training data
Why candidates think this: The drop in live AUC suggests overfitting.
Why wrong: Offline AUC remains high, indicating the model generalizes well on the validation set.
Feature extraction bug
Why candidates think this: Latency spike suggests a feature extraction issue.
Why wrong: Latency spike is a symptom, not the root cause; the real issue is the distribution shift.
📊
Monitoring to Add
  • Real-time monitoring of query length distribution
  • Alert on feature importance changes for critical features
  • Latency monitoring for feature extraction pipeline
📋
Process Change

Incorporate regular user behavior analysis into model evaluation cycles.

🏗️
Architectural Change

Design the system to automatically adapt to significant distribution shifts in input data.

💡
Key Insight
Distribution shifts in user input can drastically affect model performance if not accounted for in training and validation.

Practice

(1/5)
1. When deciding between a full factorial design and a fractional factorial design for an A/B test, which is more appropriate when the number of factors is large and why?
medium
A. Full factorial design, because it tests all possible combinations.
B. Fractional factorial design, because it is easier to interpret.
C. Fractional factorial design, because it reduces the number of tests needed.
D. Full factorial design, because it provides more accurate results.

Solution

  1. Step 1: Consider the number of factors.

    With a large number of factors, testing all combinations becomes impractical.

  2. Step 2: Eliminate wrong options.

    Full factorial is impractical for large factors, and fractional factorial is not necessarily easier to interpret.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Large factors need fewer tests. [OK]
Hint: Large factors need fewer tests. [OK]
Common Mistakes:
2. A bot detection system using device fingerprinting is failing to detect bots effectively. What is the most likely failure mode?
medium
A. The system is not considering user agent strings.
B. The system is not updating fingerprints frequently enough.
C. The system is overly reliant on IP addresses.
D. The system is using outdated behavioral models.

Solution

  1. Step 1: Identify the core issue with device fingerprinting.

    Device fingerprints need frequent updates to remain effective against evolving bots.

  2. Step 2: Eliminate wrong options.

    Behavioral models and IP reliance are not directly related to fingerprinting; user agent strings are not unique enough.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Frequent updates needed for effective fingerprinting [OK]
Hint: Frequent updates needed for effective fingerprinting [OK]
Common Mistakes:
3. What is a common failure mode when lineage tracking is not implemented in a model registry?
medium
A. Inability to reproduce model results
B. Increased model training time
C. Higher model deployment costs
D. Decreased model accuracy

Solution

  1. Step 1: Understand lineage tracking purpose

    Lineage tracking ensures that all steps and data leading to a model's creation are documented, aiding reproducibility.

  2. Step 2: Eliminate wrong options

    Increased model training time is incorrect as lineage doesn't affect training time. Higher model deployment costs is wrong because deployment costs aren't directly related to lineage. Decreased model accuracy is incorrect as accuracy isn't directly impacted by lineage tracking.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Lineage = Reproducibility [OK]
Hint: Lineage = Reproducibility [OK]
Common Mistakes:
4. In a 'People You May Know' system, what is a likely failure mode if the graph is not updated frequently?
medium
A. The system will recommend outdated connections.
B. The system will become computationally expensive.
C. The system will fail to scale with more users.
D. The system will recommend too many new connections.

Solution

  1. Step 1: Identify the impact of infrequent updates.

    Infrequent updates lead to stale data.

  2. Step 2: Evaluate options.

    Recommending too many new connections (A) is unlikely with stale data. Computational expense (B) is unrelated to update frequency. Outdated connections (C) are a direct result of stale data. Scaling issues (D) relate to system architecture, not update frequency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stale data leads to outdated recommendations. [OK]
Hint: Stale data leads to outdated recommendations. [OK]
Common Mistakes:
5. In a semantic search system using a Bi-Encoder, what is a likely failure mode if the document embeddings are not updated regularly?
medium
A. Increased computational cost during retrieval
B. Increased latency in query processing
C. Decreased retrieval accuracy due to outdated embeddings
D. Overfitting to the training data

Solution

  1. Step 1: Identify the impact of outdated embeddings

    Outdated embeddings can lead to decreased retrieval accuracy as they may not reflect current document content.

  2. Step 2: Eliminate wrong options

    Increased computational cost and latency are not directly related to outdated embeddings, and overfitting is a training issue.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Outdated embeddings = accuracy drop [check]
Hint: Outdated embeddings = accuracy drop [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard