Bird
Raised Fist0

Fine-Tuning Pipeline - SFT, RLHF, DPO Infrastructure at Scale

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Supervised Fine-Tuning (SFT) vs Reinforcement Learning from Human Feedback (RLHF)

In a large-scale language model deployment, would you choose SFT or RLHF for fine-tuning to improve user satisfaction? Justify your choice.

📊
Comparison
DimensionSupervised Fine-Tuning (SFT)Reinforcement Learning from Human Feedback (RLHF)
Training SpeedFast with labeled dataSlower due to feedback loop
Adaptability to New ScenariosLimited to labeled scenariosHigh adaptability
Resource EfficiencyHigh efficiency with fixed dataResource-intensive
User SatisfactionDependent on initial data qualityHigh with dynamic adaptation
Implementation ComplexityLow complexityHigh complexity
✅
Verdict
Default to SFT when you have a large, high-quality labeled dataset and the task environment is stable. Switch to RLHF when user preferences are dynamic and real-time feedback is available, allowing the model to adapt continuously.
💡
Key Insight
Choose SFT for efficiency with static tasks and RLHF for adaptability in dynamic environments.
⚠️
Interview Trap
❌ Wrong: Always use SFT because it's simpler and faster.
This ignores the dynamic nature of user preferences and the potential for RLHF to adapt in real-time.
✅ Consider the task environment and user feedback availability. Use SFT for stable tasks with ample labeled data, and RLHF for dynamic tasks with real-time feedback.
Supervised Fine-Tuning (SFT)
"Efficiently leverages labeled data to quickly adapt models to specific tasks."
Weaknesses
  • Requires large amounts of labeled data, which can be expensive and time-consuming to obtain.
  • May not generalize well to unseen scenarios due to overfitting on the labeled dataset.
  • Limited by the quality and diversity of the labeled data.
  • Does not incorporate real-time user feedback, potentially missing out on dynamic user preferences.
Ideal When
  • When labeled data > 100K samples is available.
  • When the task is well-defined and does not change frequently.
  • When latency < 100ms is critical and model updates are infrequent.
  • When compute resources are limited and training time is constrained.
OpenAI uses SFT to adapt GPT models to specific tasks like summarization where large labeled datasets are available.
Reinforcement Learning from Human Feedback (RLHF)
"Incorporates real-time user feedback to dynamically adapt models to user preferences."
Weaknesses
  • Complex to implement and requires a robust feedback loop.
  • Higher computational cost due to the need for continuous learning.
  • Potentially unstable if feedback is noisy or biased.
  • Longer convergence time compared to SFT due to exploration-exploitation trade-offs.
Ideal When
  • When user feedback is abundant and can be collected in real-time.
  • When the task environment is dynamic and user preferences change frequently.
  • When compute resources are ample and can support continuous training.
  • When the model needs to adapt quickly to new, unseen scenarios.
Anthropic uses RLHF to fine-tune language models for better alignment with human values and preferences.
🐍
Supervised Fine-Tuning (SFT) - Example
▼ Show
Python
from transformers import Trainer, TrainingArguments

model = ...  # Pre-trained model
train_dataset = ...  # Labeled dataset

training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()
🐍
Reinforcement Learning from Human Feedback (RLHF) - Example
▼ Show
Python
import gym
from stable_baselines3 import PPO

env = gym.make('CustomEnv-v0')  # Custom environment with human feedback

model = PPO('MlpPolicy', env, verbose=1)

model.learn(total_timesteps=10000)

obs = env.reset()
for _ in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = env.step(action)
    if dones:
        obs = env.reset()
✅
Use Supervised Fine-Tuning (SFT) When
  • •When labeled data > 100K samples is available and high quality.
  • •When the task is stable and does not require frequent updates.
  • •When compute resources are limited and training time is constrained.
  • •When latency < 100ms is critical for the application.
✅
Use Reinforcement Learning from Human Feedback (RLHF) When
  • •When real-time user feedback is available and abundant.
  • •When the task environment is dynamic and user preferences change frequently.
  • •When compute resources are ample and can support continuous training.
  • •When the model needs to adapt quickly to new, unseen scenarios.
↳
Nuances
  • For rapidly changing environments, RLHF can outperform SFT despite higher complexity.
  • In scenarios with limited labeled data, RLHF can leverage user feedback to improve performance.
  • SFT may be preferred in regulated industries where data labeling is controlled and consistent.
  • RLHF can introduce bias if feedback is not representative of the target user base.

Practice

(1/5)
1. For a feed ranking system, which metric should be optimized to ensure users see the most relevant content?
easy
A. Click-through rate (CTR)
B. Page load time
C. Number of ads displayed
D. Total time spent on platform

Solution

  1. Step 1: Understand the goal of relevance

    CTR measures how often users click on posts, indicating relevance.

  2. Step 2: Eliminate wrong options

    Page load time affects performance, not relevance. Number of ads and total time spent are not direct indicators of content relevance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    CTR indicates relevance. [OK]
Hint: CTR indicates relevance. [OK]
Common Mistakes:
2. In the context of a Guardrails System for LLMs, which component is primarily responsible for detecting inappropriate content in user-generated text?
easy
A. Data augmentation module
B. Content filtering module
C. Model training module
D. User feedback loop

Solution

  1. Step 1: Identify the role of each component

    The content filtering module is designed to detect and filter inappropriate content.

  2. Step 2: Eliminate wrong options

    Data augmentation is for enhancing training data, model training is for learning patterns, and user feedback loop is for improving system iteratively.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Content filtering detects inappropriate content. [OK]
Hint: Content filtering detects inappropriate content. [OK]
Common Mistakes:
3. For a semantic search system using a Bi-Encoder, which metric is most appropriate to evaluate the retrieval performance?
easy
A. Accuracy
B. Precision@K
C. Mean Squared Error
D. F1 Score

Solution

  1. Step 1: Understand retrieval evaluation metrics

    Precision@K measures the relevance of the top K retrieved documents, which is crucial for retrieval tasks.

  2. Step 2: Eliminate wrong options

    Accuracy and F1 Score are more suited for classification tasks, Mean Squared Error is for regression.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Precision@K = retrieval relevance [check]
Hint: Precision@K = retrieval relevance [check]
Common Mistakes:
4. In the context of a Two-Tower Model for Retrieval, what is the primary role of the user tower?
easy
A. To generate item embeddings based on user interactions.
B. To generate user embeddings based on user features.
C. To compute similarity scores between users and items.
D. To directly predict user-item interaction scores.

Solution

  1. Step 1: Understand the user tower's role

    The user tower is responsible for generating embeddings that represent users based on their features.

  2. Step 2: Eliminate wrong options

    To generate item embeddings based on user interactions. is incorrect because item embeddings are generated by the item tower. To compute similarity scores between users and items. is incorrect because similarity scores are computed after embeddings are generated. To directly predict user-item interaction scores. is incorrect because the model predicts interaction scores using both towers' outputs.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    User tower generates user embeddings. [OK]
Hint: User tower generates user embeddings. [OK]
Common Mistakes:
5. A session-based recommendation model is underperforming. Which of the following is a likely failure mode specific to sequential modeling?
medium
A. Ignoring the order of interactions.
B. Data leakage from test to train set.
C. Using a too small learning rate.
D. Overfitting due to too many features.

Solution

  1. Step 1: Identify sequential modeling issues

    Ignoring the order of interactions can lead to poor performance in sequential models.

  2. Step 2: Eliminate generic issues

    Overfitting and data leakage are general issues; learning rate affects convergence speed, not sequence handling.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Order matters in sequences. [OK]
Hint: Order matters in sequences. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard