Bird
Raised Fist0

The Multimodal Mismatch

SYSTEM-DESIGNseniorInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Multimodal Mismatch
Started last Friday and has been ongoing for 3 days.

A vision-language model used for product recommendations on an e-commerce platform, serving 5M requests per day.

Symptom
Recommendation diversity collapsed, AUC dropped from 0.85 to 0.72 in live traffic, while offline AUC remained stable.
📡
Signals
metric
Live AUC dropped significantly while offline AUC remained stable.
Live AUC dropped from 0.85 to 0.72; offline AUC stable at 0.89.
distribution
Distribution of image features in live traffic compared to training data.
Live image feature distribution shifted significantly from training data.
feature_importance
Feature importance for 'image_color_histogram' decreased.
Importance dropped from 0.15 to 0.05 after last Friday's deploy.
latency
Real-time feature extraction latency spiked.
P99 latency increased from 80ms to 150ms.
log
Error logs showing increased timeouts in feature extraction service.
Timeout errors increased by 30% since last Friday.
🔬
Hypotheses
Image feature distribution shift is causing the model to perform poorly on live data.
Supporting Signals
distributionfeature_importance
Verify: Compare the distribution of live image features with the training set.
import matplotlib.pyplot as plt; plt.hist(live_features, bins=50, alpha=0.5, label='Live'); plt.hist(training_features, bins=50, alpha=0.5, label='Training'); plt.legend(); plt.show()
Increased latency in feature extraction is causing timeouts and incomplete data processing.
Supporting Signals
latencylog
Verify: Check the logs for timeout errors and correlate with latency spikes.
import pandas as pd; logs = pd.read_csv('logs.csv'); timeout_errors = logs[logs['error'] == 'timeout']; timeout_errors['timestamp'].plot(kind='hist')
🎯
Root Cause
A recent deployment changed the image feature extraction pipeline, causing a distribution shift and increased latency.
Why It Happened

The deployment introduced a new image processing library that altered feature extraction characteristics and increased processing time.

🔧
Fix Strategy
ImmediateRollback to the previous version of the image processing library.
Short-termOptimize the new image processing library to match the previous feature distribution and reduce latency.
Long-termImplement a canary deployment strategy for feature extraction changes to detect issues before full rollout.
🐍
Python Fix
Python
def rollback_feature_extraction():
    # Code to rollback to previous library version
    subprocess.run(['pip', 'install', 'old-image-lib==1.0.0'])
    restart_feature_extraction_service()
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The stable offline AUC suggests the model is not generalizing well.
Why wrong: The issue is not with the model's generalization but with the input feature distribution shift.
A bug in the recommendation logic caused the diversity collapse.
Why candidates think this: The collapse in recommendation diversity suggests a logic error.
Why wrong: The root cause is related to feature extraction, not recommendation logic.
📊
Monitoring to Add
  • Real-time monitoring of feature distribution shifts using KL divergence.
  • Latency monitoring with alerts for P99 spikes above 100ms.
  • Error rate monitoring for feature extraction service.
📋
Process Change

Introduce mandatory A/B testing for all major pipeline changes.

🏗️
Architectural Change

Adopt a microservices architecture for feature extraction to isolate and scale components independently.

💡
Key Insight
Feature extraction changes can silently alter input distributions, causing significant online performance degradation even when offline metrics appear stable.

Practice

(1/5)
1. For an ad click-through rate prediction model, which metric is most appropriate to optimize for business value?
easy
A. Recall
B. Precision
C. AUC-ROC
D. Accuracy

Solution

  1. Step 1: Understand business objectives.

    CTR prediction models need to balance true positive rate and false positive rate.

  2. Step 2: Eliminate wrong options.

    Accuracy (A) doesn't account for class imbalance. Precision (B) and Recall (C) focus on one aspect of performance, not the trade-off.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    AUC-ROC balances trade-offs in CTR. [OK]
Hint: AUC-ROC balances trade-offs in CTR. [OK]
Common Mistakes:
2. In the context of agent system design, which component is primarily responsible for deciding the sequence of actions to achieve a goal?
easy
A. Memory Architecture
B. Planning
C. Tool Use
D. Data Preprocessing

Solution

  1. Step 1: Identify the role of each component

    Planning is responsible for deciding the sequence of actions to achieve a goal.

  2. Step 2: Eliminate wrong options

    Memory Architecture is for storing information, Tool Use is for executing actions, and Data Preprocessing is unrelated to action sequencing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Planning = action sequence decision [OK]
Hint: Planning decides action sequences. [OK]
Common Mistakes:
3. In the context of a 'People You May Know' feature using graph-based link prediction, which algorithm is most suitable for predicting new connections based on node similarity?
easy
A. Node2Vec
B. Collaborative Filtering
C. K-Means Clustering
D. Decision Trees

Solution

  1. Step 1: Identify the task.

    The task is to predict new connections based on node similarity in a graph.

  2. Step 2: Evaluate options.

    K-Means Clustering (A) is for clustering, not link prediction. Collaborative Filtering (B) is for recommendation systems, not graph-based predictions. Decision Trees (D) are for classification/regression, not graph-based link prediction. Node2Vec (C) is designed for learning node embeddings in graphs, making it suitable for link prediction.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Node2Vec for graph embeddings. [OK]
Hint: Node2Vec is for graph embeddings. [OK]
Common Mistakes:
4. When transitioning from logistic regression to DLRM for CTR prediction, which trade-off is most critical to consider?
medium
A. Model interpretability vs. prediction accuracy
B. Training speed vs. model complexity
C. Feature engineering effort vs. model performance
D. Data preprocessing time vs. inference latency

Solution

  1. Step 1: Identify key differences between models.

    DLRM offers higher accuracy but is less interpretable than logistic regression.

  2. Step 2: Eliminate wrong options.

    Training speed (B) and preprocessing time (D) are less critical than understanding model decisions. Feature engineering (C) is important but not the primary trade-off.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DLRM: accuracy vs. interpretability trade-off. [OK]
Hint: DLRM: accuracy vs. interpretability trade-off. [OK]
Common Mistakes:
5. In a recommendation system designed to avoid filter bubbles, what is a likely failure mode if the diversity re-ranking layer is not properly tuned?
medium
A. Loss of user engagement
B. Increased computational cost
C. Overfitting to user preferences
D. Increased latency in recommendations

Solution

  1. Step 1: Identify the impact of poor tuning

    Poor tuning can lead to irrelevant recommendations, reducing user engagement.

  2. Step 2: Eliminate wrong options

    Increased computational cost (A) and latency (D) are technical issues, not directly related to user engagement. Overfitting (C) is more about personalization than diversity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Poor tuning reduces engagement due to irrelevant content [OK]
Hint: Poor tuning reduces engagement due to irrelevant content [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard