Bird
Raised Fist0

Multimodal System Design - Vision-Language Models at Scale

SYSTEM-DESIGNseniorInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Multimodal System Design - Vision-Language Models at Scale
Designing scalable systems for vision-language models
system-designseniorFreq: mediumGenAI Systems
Prerequisites:Transformer ArchitectureComputer Vision - Object DetectionEmbeddings
🎯
Problem Framing
Business Objective

Maximize user engagement by providing relevant multimodal content recommendations.

ML Objective

Maximize P(engagement|image, text, user context) using a vision-language model.

Constraints
  • P99 latency < 100ms for inference
  • High throughput of 1000 QPS
  • Limited labeled data for new content
  • Compliance with data privacy regulations
  • Cold start problem for new users and content
Clarifying Questions
  • What is the expected scale of data ingestion per day?
  • How frequently does the content catalog update?
  • What are the privacy constraints on user data?
  • What is the acceptable trade-off between latency and accuracy?
  • How is user engagement currently measured?
Out of Scope
✗ Non-vision or non-language modalities
✗ Backend infrastructure unrelated to ML
✗ User interface design
📊
Offline Metrics
PR-AUC: To handle class imbalance in engagement prediction.
BLEU score: For evaluating the quality of text generation in language tasks.
📈
Online Metrics
CTR: Measures the effectiveness of recommendations in driving clicks.
Session duration: Indicates user engagement with the recommended content.
⚖️
Trade-offs

Offline metrics like PR-AUC may not directly correlate with online metrics like CTR due to differences in user behavior and model assumptions.

💼
Business Metric

Increase in average session duration per user.

🗄️
Data Sources
  • User interaction logs (clicks, views) - real-time
  • Content metadata (images, text descriptions) - updated daily
  • User profiles (demographics, preferences) - updated weekly
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, time spent) to derive engagement labels.

Data Volume

Approximately 10 million interactions per day, 5 TB of data per month.

Freshness

Features should be updated in near real-time to reflect the latest user interactions.

⚠️
Data Challenges
  • Sparse labels for new content
  • Position bias in user interactions
  • Survivorship bias in engagement data
  • Cold start for new users and content
🔧
Key Features
embedding
Image Embeddings
Capture visual content features for multimodal understanding.
embedding
Text Embeddings
Capture semantic meaning of content descriptions.
categorical
User Interaction History
Provide context on user preferences and engagement patterns.
numerical
Content Popularity
Indicate general user interest in content.
categorical
Time of Day
Capture temporal patterns in user engagement.
⚡
Feature Engineering Notes
Real-time vs Batch

User interaction history and content popularity need real-time updates, while image and text embeddings can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_image_embedding(image):
    model = load_pretrained_model()
    embedding = model.extract_features(image)
    return embedding
🤖
Model Selection
Baseline

Logistic Regression using pre-computed embeddings.

Proposed Model

Vision-Language Transformer (e.g., CLIP) for joint image-text understanding.

Handles both vision and language data effectively, leveraging pre-trained embeddings to reduce training time and improve accuracy.
✗
Alternatives Rejected
CNN + RNN: Rejected due to inability to capture complex interactions between modalities.
XGBoost: Rejected due to high latency and inability to handle multimodal data natively.
🏋️
Training Strategy

Offline training with periodic updates using new interaction data to adapt to changing user preferences.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput1000 QPS
ArchitectureTwo-stage (retrieval + ranking) to efficiently handle large-scale data and reduce latency.
CachingCache popular content embeddings with a TTL of 1 hour to reduce computation load.
FallbackUse rule-based recommendations based on content popularity when the model is unavailable.
📡
Data Drift Signals
Shift in user interaction distribution
Change in content feature distributions
Increase in missing data rates
📉
Model Drift Signals
Decrease in PR-AUC over time
Shift in prediction score distribution
Increase in prediction latency
🔔
Alerting & Retraining
Retraining Trigger

PR-AUC drops below 0.75 or significant drift in feature distributions.

Alerting Strategy

On-call engineer paged for significant metric drops, logs for prediction errors, auto-recovery for minor drifts.

🎯
Interview Angles
  • How to handle multimodal data efficiently?
  • Trade-offs between model complexity and latency?
  • Strategies for dealing with cold start problems?
⚠️
Common Mistakes
Ignoring latency constraints in model selection
Overfitting to offline metrics without considering online performance
Neglecting data privacy and compliance issues
Failing to address cold start challenges
📌
Key Decisions
  • Choosing a vision-language transformer for multimodal understanding
  • Implementing a two-stage serving architecture for scalability
  • Using implicit feedback for labeling to handle sparse data

Practice

(1/5)
1. For an ad auction system aiming to maximize revenue, which metric should be optimized?
easy
A. Click-through Rate (CTR)
B. Conversion Rate
C. Ad Impressions
D. Expected Revenue

Solution

  1. Step 1: Align metric with objective

    To maximize revenue, the system should optimize for Expected Revenue, which considers both CTR and bid amounts.

  2. Step 2: Eliminate wrong options

    • Click-through Rate (CTR): Measures engagement, not directly linked to revenue.
    • Conversion Rate: Focuses on post-click actions, not directly on revenue.
    • Ad Impressions: Measures visibility, not revenue.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Optimize Expected Revenue for revenue goals. [OK]
Hint: Optimize Expected Revenue for revenue goals. [OK]
Common Mistakes:
2. In a real-time fraud detection system, which component is responsible for scoring transactions as they occur?
easy
A. Real-time scoring engine
B. Offline model training
C. Data warehouse
D. Batch processing engine

Solution

  1. Step 1: Identify the component for real-time scoring.

    The real-time scoring engine is designed to evaluate transactions as they occur.

  2. Step 2: Eliminate wrong options.

    Batch processing engine (A) is for batch jobs, Data warehouse (C) is for storage, and Offline model training (D) is for model development.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Real-time scoring needs a real-time engine. [OK]
Hint: Real-time scoring needs a real-time engine. [OK]
Common Mistakes:
3. In the context of intent classification, which of the following techniques is most suitable for handling imbalanced datasets?
easy
A. Apply SMOTE (Synthetic Minority Over-sampling Technique)
B. Use a simple logistic regression model
C. Use a linear SVM without any modifications
D. Ignore the imbalance and focus on feature engineering

Solution

  1. Step 1: Identify the challenge of imbalanced datasets.

    Imbalanced datasets can lead to biased models that favor the majority class.

  2. Step 2: Evaluate each option.

    Use a simple logistic regression model: Logistic regression without addressing imbalance will likely be biased. Use a linear SVM without any modifications: Linear SVM also suffers from imbalance without adjustments. Ignore the imbalance and focus on feature engineering: Ignoring imbalance is not a solution. Apply SMOTE (Synthetic Minority Over-sampling Technique): SMOTE is specifically designed to handle imbalance by generating synthetic samples.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    SMOTE for imbalance handling. [OK]
Hint: SMOTE is key for imbalanced datasets. [OK]
Common Mistakes:
4. When transitioning from logistic regression to DLRM for CTR prediction, which trade-off is most critical to consider?
medium
A. Model interpretability vs. prediction accuracy
B. Training speed vs. model complexity
C. Feature engineering effort vs. model performance
D. Data preprocessing time vs. inference latency

Solution

  1. Step 1: Identify key differences between models.

    DLRM offers higher accuracy but is less interpretable than logistic regression.

  2. Step 2: Eliminate wrong options.

    Training speed (B) and preprocessing time (D) are less critical than understanding model decisions. Feature engineering (C) is important but not the primary trade-off.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DLRM: accuracy vs. interpretability trade-off. [OK]
Hint: DLRM: accuracy vs. interpretability trade-off. [OK]
Common Mistakes:
5. As the number of concurrent users increases tenfold, which component of the LLM serving infrastructure is most likely to become a bottleneck first?
hard
A. Network bandwidth
B. Batching mechanism
C. Speculative Decoding
D. KV Cache

Solution

  1. Step 1: Consider the impact of scaling.

    Network bandwidth is often the first bottleneck as user requests increase.

  2. Step 2: Eliminate wrong options.

    Batching can handle more requests efficiently. KV Cache scales with memory, not users. Speculative Decoding affects processing, not network.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Network bandwidth limits scale first. [OK]
Hint: Network bandwidth limits scale first. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard