Bird
Raised Fist0

ML Platform - Feature Store, Training, Serving Infra, End-to-End

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
ML Platform - Feature Store, Training, Serving Infra, End-to-End
Design a robust ML platform with integrated feature store, training, and serving infrastructure.
system-designseniorFreq: very_highML Platforms
Prerequisites:MLOps CI/CDFeature EngineeringContinual Learning vs Periodic Retraining
🎯
Problem Framing
Business Objective

Maximize user engagement and retention through personalized content recommendations.

ML Objective

Maximize P(click|user, content, context) for content recommendations.

Constraints
  • P99 latency < 100ms for serving recommendations
  • Data privacy compliance with GDPR
  • Scalable to 1 million users daily
  • Cold start problem for new users and content
  • Real-time feature updates for dynamic user behavior
Clarifying Questions
  • What is the expected scale of user interactions per day?
  • How often does the content catalog update?
  • What are the privacy and compliance requirements?
  • What is the acceptable trade-off between latency and accuracy?
  • How is user feedback collected and integrated into the system?
Out of Scope
✗ Non-personalized content delivery
✗ User interface design
✗ Backend infrastructure unrelated to ML
📊
Offline Metrics
PR-AUC: Suitable for imbalanced click data where positive interactions are rare.
NDCG@10: Focuses on ranking quality by emphasizing top results.
📈
Online Metrics
CTR: Directly measures user engagement with recommendations.
Revenue per session: Evaluates the business impact of recommendations.
⚖️
Trade-offs

High offline PR-AUC may not always translate to high online CTR due to real-world noise and user behavior variability.

💼
Business Metric

Overall increase in user engagement and retention rates.

🗄️
Data Sources
  • User interaction logs (clicks, views) - real-time
  • Content metadata (categories, tags) - updated daily
  • User profiles (demographics, preferences) - updated weekly
  • Third-party data (trending topics) - updated hourly
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, time spent) to infer preferences.

Data Volume

Approximately 10 million interactions per day, 5 TB of data per month.

Freshness

Features must be updated in real-time to reflect current user behavior.

⚠️
Data Challenges
  • Sparse labels due to infrequent clicks
  • Position bias affecting click-through rates
  • Cold start for new users and content
  • Data privacy and compliance with regulations
🔧
Key Features
numerical
User activity score
Captures recent user engagement level.
numerical
Content popularity
Reflects general interest in content.
embedding
User-content interaction history
Encodes past interactions for personalization.
categorical
Time of day
Accounts for temporal patterns in user behavior.
categorical
Device type
Differentiates user experience across devices.
⚡
Feature Engineering Notes
Real-time vs Batch

User activity score and content popularity need real-time computation, while user-content interaction history can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_user_activity_score(user_id, interactions):
    recent_interactions = filter_recent(interactions, days=7)
    return len(recent_interactions) / 7
🤖
Model Selection
Baseline

Popularity-based model using content popularity scores.

Proposed Model

Two-tower neural network model for capturing user-content interactions.

The two-tower model efficiently handles large-scale user-item interactions and allows pre-computation of item embeddings, reducing online latency.
✗
Alternatives Rejected
Logistic Regression: Rejected due to inability to capture complex user-item interactions.
XGBoost: Rejected due to latency and memory overhead incompatible with real-time requirements.
🏋️
Training Strategy

Hybrid approach with offline training for base model and online learning for real-time updates.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage (retrieval + ranking) to efficiently handle large candidate sets and refine top results.
CachingCache popular content recommendations with a TTL of 5 minutes to reduce computation load.
FallbackUse rule-based recommendations based on content popularity when the model is unavailable.
📡
Data Drift Signals
Feature distribution drift in user activity scores
Label shift in click-through rates
Content popularity distribution changes
📉
Model Drift Signals
AUC degradation over time
Shift in prediction score distribution
Increase in prediction error rates
🔔
Alerting & Retraining
Retraining Trigger

Retrain when AUC drops by more than 5% over a month.

Alerting Strategy

Page on-call engineer if prediction error rate exceeds 10% or if data drift is detected beyond thresholds.

🎯
Interview Angles
  • How to handle cold start problems for new users and content?
  • What strategies ensure compliance with data privacy regulations?
  • How to balance latency and accuracy in real-time systems?
⚠️
Common Mistakes
Ignoring the impact of data privacy regulations on feature engineering.
Overlooking the need for real-time feature updates.
Choosing complex models without considering latency constraints.
Failing to define clear retraining triggers based on model performance.
📌
Key Decisions
  • Selecting a two-tower model for efficient user-item interaction handling.
  • Implementing a feature store for real-time and batch feature management.
  • Designing a two-stage serving architecture to optimize latency and throughput.

Practice

(1/5)
1. In the context of device fingerprinting for bot detection, which component is primarily responsible for identifying unique devices?
easy
A. IP address tracking
B. User agent string analysis
C. Cookie storage
D. Hardware and software configuration analysis

Solution

  1. Step 1: Identify the key component for device uniqueness.

    Device fingerprinting relies on unique hardware and software configurations to identify devices.

  2. Step 2: Eliminate wrong options.

    IP addresses and cookies can change or be shared; user agent strings are not unique enough.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Device uniqueness = hardware/software config [OK]
Hint: Device uniqueness = hardware/software config [OK]
Common Mistakes:
2. An agent system fails to adapt to new tasks despite having a learning component. What is the most likely failure mode?
medium
A. Inadequate tool use integration
B. Insufficient memory capacity
C. Overfitting to initial tasks
D. Poor initial planning strategy

Solution

  1. Step 1: Identify symptoms

    The system fails to adapt, indicating a learning issue.

  2. Step 2: Analyze potential causes

    Overfitting to initial tasks prevents generalization to new tasks, while other options relate to different issues.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Adaptation failure = Overfitting issue [OK]
Hint: Adaptation failure often means overfitting. [OK]
Common Mistakes:
3. When deciding between a centralized model registry and a decentralized one, which is a key consideration?
medium
A. Decentralized registries are easier to scale
B. Decentralized registries offer better version control
C. Centralized registries are always faster
D. Centralized registries simplify governance and compliance

Solution

  1. Step 1: Consider governance and compliance

    Centralized registries simplify governance and compliance by having a single point of control.

  2. Step 2: Eliminate wrong options

    Centralized registries are always faster is incorrect as speed depends on implementation. Decentralized registries offer better version control is wrong because version control is not inherently better in decentralized systems. Decentralized registries are easier to scale is incorrect as scaling depends on architecture, not centralization.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Centralized = Simplified governance [OK]
Hint: Centralized = Simplified governance [OK]
Common Mistakes:
4. As a content-based filtering system scales to 10x more items, which component is most likely to become a bottleneck?
hard
A. Item embedding computation
B. User profile storage
C. Recommendation generation
D. User-item interaction logging

Solution

  1. Step 1: Analyze scaling impact

    Scaling to more items increases the demand for computing and updating item embeddings, making it a potential bottleneck.

  2. Step 2: Eliminate wrong options

    A: User profile storage scales with users, not items. C: Recommendation generation is more affected by user scale. D: Interaction logging is more about user activity than item count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    More items strain embedding computation. [OK]
Hint: More items strain embedding computation. [OK]
Common Mistakes:
5. As the number of LLMs evaluated increases tenfold, which component of the evaluation system is most likely to fail first?
hard
A. Human evaluator availability
B. Benchmark task diversity
C. Data storage capacity
D. LLM-as-Judge processing power

Solution

  1. Step 1: Analyze scalability of components

    Human evaluators are a bottleneck as scaling requires more human resources.

  2. Step 2: Eliminate wrong options

    Data storage and processing power can be scaled with infrastructure, and task diversity is not a direct scalability issue.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Human evaluators are the bottleneck in scaling. [OK]
Hint: Human evaluators are the bottleneck in scaling. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard