Bird
Raised Fist0

Fine-Tuning Pipeline - SFT, RLHF, DPO Infrastructure at Scale

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Fine-Tuning Pipeline - SFT, RLHF, DPO Infrastructure at Scale
Design scalable infrastructure for fine-tuning large language models with SFT, RLHF, and DPO.
system-designseniorFreq: very_highGenAI Systems
Prerequisites:Transfer Learning and Fine-TuningMLOps CI/CD
🎯
Problem Framing
Business Objective

Maximize user engagement and satisfaction with personalized content recommendations.

ML Objective

Optimize the language model's ability to generate contextually relevant and user-preferred responses.

Constraints
  • Latency SLA: P99 < 100ms for response generation
  • Data privacy regulations (GDPR, CCPA)
  • Scalability to handle 10 million users daily
  • Model interpretability for compliance
  • Limited labeled data for RLHF
Clarifying Questions
  • What is the expected scale of user interactions per day?
  • Are there specific privacy regulations we need to comply with?
  • What is the current infrastructure for model deployment?
  • How frequently should the model be retrained?
  • What fallback mechanisms are in place if the model fails?
Out of Scope
✗ Building the initial language model from scratch
✗ Non-ML related user interface design
✗ Backend infrastructure unrelated to ML serving
📊
Offline Metrics
BLEU score for evaluating the quality of generated text
PR-AUC for assessing the model's ability to rank preferred responses
📈
Online Metrics
CTR (Click-Through Rate) for content engagement
User satisfaction score from feedback surveys
⚖️
Trade-offs

Offline metrics like BLEU may not correlate directly with user satisfaction, which is better captured by online metrics like CTR and user feedback.

💼
Business Metric

Overall user engagement measured by session length and return visits.

🗄️
Data Sources
  • User interaction logs (textual data, real-time)
  • Feedback surveys (numerical ratings, T-1 day)
  • Content metadata (categorical, T-1 day)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions and explicit feedback from surveys.

Data Volume

Approximately 500 million interactions per day, 10 TB/month.

Freshness

Real-time for interaction logs, T-1 day for feedback surveys.

⚠️
Data Challenges
  • Sparse explicit feedback
  • Position bias in interaction logs
  • Cold start for new users
  • Survivorship bias in feedback
🔧
Key Features
embedding
User interaction history
Captures user preferences and context.
categorical
Content metadata
Provides context for content relevance.
numerical
Feedback score
Directly measures user satisfaction.
numerical
Session duration
Indicator of engagement level.
numerical
Time of interaction
Temporal patterns in user behavior.
⚡
Feature Engineering Notes
Real-time vs Batch

User interaction history and time of interaction need real-time computation, while content metadata and feedback scores can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
import numpy as np

def compute_user_embedding(interactions):
    # Example function to compute user embeddings
    return np.mean([interaction.embedding for interaction in interactions], axis=0)
🤖
Model Selection
Baseline

Logistic regression on user interaction features.

Proposed Model

Transformer-based model fine-tuned with RLHF and DPO.

Transformers are state-of-the-art for language tasks and can be fine-tuned to incorporate user preferences effectively.
✗
Alternatives Rejected
RNNs: Rejected due to inability to handle long-range dependencies efficiently.
XGBoost: Rejected because it cannot capture complex language patterns as effectively as transformers.
🏋️
Training Strategy

Hybrid approach with offline pre-training and online fine-tuning using RLHF.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureSingle-stage architecture to minimize latency, leveraging pre-computed embeddings.
CachingCache recent user embeddings with a TTL of 5 minutes, invalidate on new interactions.
FallbackRule-based fallback using cached results if the model is unavailable.
📡
Data Drift Signals
Shift in user interaction distribution
Change in feedback score distribution
Variation in content metadata usage
📉
Model Drift Signals
Decrease in BLEU score over time
Drop in CTR
Increase in user dissatisfaction reports
🔔
Alerting & Retraining
Retraining Trigger

CTR drops by more than 5% over a week.

Alerting Strategy

Page on-call engineer if CTR drops by 10% in a day, log all prediction errors, auto-recover by rolling back to the last stable model.

🎯
Interview Angles
  • How do you handle data privacy in this system?
  • What are the trade-offs between offline and online metrics?
  • How do you ensure the model remains performant over time?
⚠️
Common Mistakes
Ignoring data privacy regulations
Over-relying on offline metrics without considering online performance
Neglecting to plan for model drift and retraining
📌
Key Decisions
  • Choosing a transformer-based model for its language capabilities
  • Implementing a hybrid training strategy to balance pre-training and fine-tuning
  • Designing a caching strategy to meet latency requirements

Practice

(1/5)
1. In the context of a Guardrails System for LLMs, which component is primarily responsible for detecting inappropriate content in user-generated text?
easy
A. Data augmentation module
B. Content filtering module
C. Model training module
D. User feedback loop

Solution

  1. Step 1: Identify the role of each component

    The content filtering module is designed to detect and filter inappropriate content.

  2. Step 2: Eliminate wrong options

    Data augmentation is for enhancing training data, model training is for learning patterns, and user feedback loop is for improving system iteratively.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Content filtering detects inappropriate content. [OK]
Hint: Content filtering detects inappropriate content. [OK]
Common Mistakes:
2. When choosing between Isolation Forest and Autoencoder for anomaly detection, which scenario favors using an Autoencoder?
medium
A. When the dataset is small and labeled.
B. When interpretability of the model is crucial.
C. When the anomalies are expected to be in high-dimensional data.
D. When the dataset is large and unlabeled.

Solution

  1. Step 1: Consider the strengths of Autoencoders.

    Autoencoders are particularly effective for high-dimensional data due to their ability to learn complex representations.

  2. Step 2: Eliminate wrong options.

    When the dataset is small and labeled. is incorrect because Autoencoders require more data to train effectively. When interpretability of the model is crucial. is incorrect because Autoencoders are less interpretable than Isolation Forests. When the dataset is large and unlabeled. is incorrect because Autoencoders are computationally intensive and may not be ideal for very large datasets.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Autoencoder = high-dimensional data strength [OK]
Hint: Autoencoder = high-dimensional data strength [OK]
Common Mistakes:
3. When designing a session-based recommendation system, why might you choose a Transformer model over an RNN?
medium
A. RNNs require less data preprocessing.
B. Transformers are faster for small datasets.
C. RNNs are more interpretable.
D. Transformers handle long-range dependencies better.

Solution

  1. Step 1: Compare model capabilities

    Transformers are known for effectively capturing long-range dependencies due to their attention mechanism.

  2. Step 2: Eliminate incorrect reasons

    Transformers are not necessarily faster for small datasets; RNNs are not more interpretable or require less preprocessing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Transformers for long-range dependencies. [OK]
Hint: Transformers for long-range dependencies. [OK]
Common Mistakes:
4. As the number of users and ads increases tenfold in an ad auction system, which component is most likely to fail first?
hard
A. User Data Storage
B. Ad Ranking Module
C. Bid Management System
D. Network Bandwidth

Solution

  1. Step 1: Consider scale impact

    With a tenfold increase in users and ads, the User Data Storage is likely to face capacity issues first due to the sheer volume of data.

  2. Step 2: Eliminate wrong options

    • Ad Ranking Module: Computationally intensive but not storage-bound.
    • Bid Management System: Handles bids, not directly affected by user volume.
    • Network Bandwidth: Affects data transfer, but storage is a more immediate bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Storage fails first with data volume increase. [OK]
Hint: Storage fails first with data volume increase. [OK]
Common Mistakes:
5. As the number of images in a visual search system scales up by 10x, which component is most likely to become a bottleneck first?
hard
A. Approximate Nearest Neighbor (ANN) index
B. Network bandwidth
C. Serving layer
D. Image embedding model

Solution

  1. Step 1: Analyze scaling impact

    The ANN index is likely to become a bottleneck as it handles search queries, which increase with more data.

  2. Step 2: Eliminate wrong options

    Embedding model scales with compute, serving layer handles requests, and network bandwidth is less likely to be the first bottleneck.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ANN index struggles first with scale. [OK]
Hint: ANN index struggles first with scale. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard