Bird
Raised Fist0

The Fine-Tuning Fiasco

SYSTEM-DESIGNseniorInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Fine-Tuning Fiasco
Started last Friday, ongoing for 3 days.

A fine-tuning pipeline for a large-scale language model using SFT, RLHF, and DPO serving 10M requests/day for a conversational AI platform.

Symptom
Model achieves 0.92 AUC offline but drops to 0.75 AUC on live traffic. Recommendation diversity collapsed, with top 5 responses being nearly identical across different contexts.
📡
Signals
metric
Live AUC dropped significantly compared to offline AUC.
Live AUC is 0.75, offline AUC is 0.92.
distribution
Distribution of model outputs shows high concentration on a few responses.
Top 5 responses account for 80% of all outputs.
feature_importance
Feature 'user_last_active_days' importance dropped significantly.
Importance dropped from 0.18 to 0.02 after last Friday's deploy.
latency
Inference latency spiked for certain requests.
P99 latency increased from 80ms to 150ms.
log
Error logs show increased frequency of 'timeout' errors during inference.
Timeout errors increased by 30% since last Friday.
🔬
Hypotheses
The model is overfitting to recent data patterns due to a data pipeline issue.
Supporting Signals
Live AUC dropFeature importance change
Verify: Check if recent data has a distribution shift compared to training data.
import pandas as pd
import matplotlib.pyplot as plt
recent_data = pd.read_csv('recent_data.csv')
training_data = pd.read_csv('training_data.csv')
recent_data['feature'].hist(alpha=0.5, label='Recent')
training_data['feature'].hist(alpha=0.5, label='Training')
plt.legend()
plt.show()
Inference latency increase is causing timeouts, affecting live AUC.
Supporting Signals
Latency spikeTimeout errors
Verify: Analyze server logs for correlation between latency spikes and AUC drops.
import pandas as pd
import matplotlib.pyplot as plt
latency_data = pd.read_csv('latency_logs.csv')
auc_data = pd.read_csv('auc_logs.csv')
plt.plot(latency_data['timestamp'], latency_data['latency'], label='Latency')
plt.plot(auc_data['timestamp'], auc_data['auc'], label='AUC')
plt.legend()
plt.show()
🎯
Root Cause
A recent deployment introduced a bug in the data pipeline, causing a distribution shift in the training data.
Why It Happened

An upstream data source changed its schema, leading to incorrect feature extraction during training.

🔧
Fix Strategy
ImmediateRollback to the last known good model version.
Short-termCorrect the data pipeline to handle the new schema and retrain the model.
Long-termImplement schema validation and automated alerts for data pipeline changes.
🐍
Python Fix
Python
def validate_schema(data):
    expected_columns = ['col1', 'col2', 'col3']
    if set(data.columns) != set(expected_columns):
        raise ValueError('Schema mismatch!')

# Apply schema validation before processing
data = pd.read_csv('new_data.csv')
validate_schema(data)
# Proceed with data processing
✗
Common Wrong Diagnoses
▼ Show
The model is simply overfitting to the training data.
Why candidates think this: The offline AUC is high, suggesting overfitting.
Why wrong: Overfitting would not cause a sudden drop in live AUC without a change in data distribution.
The latency spike is the primary cause of AUC drop.
Why candidates think this: Increased latency could lead to timeouts and affect performance.
Why wrong: Latency issues are a symptom of the underlying data distribution problem.
📊
Monitoring to Add
  • Real-time AUC monitoring on live traffic.
  • Distribution shift detection between training and live data.
  • Schema validation alerts for data pipeline.
  • Latency monitoring with alert thresholds.
📋
Process Change

Introduce a mandatory data pipeline review process for all upstream changes.

🏗️
Architectural Change

Adopt a feature store to decouple feature extraction from raw data ingestion, ensuring consistency.

💡
Key Insight
Data pipeline integrity is crucial; even minor upstream changes can cascade into significant model performance issues.

Practice

(1/5)
1. In a search ranking system, which component is primarily responsible for determining the relevance of a document to a query?
easy
A. Query parser
B. Ranking algorithm
C. Indexing module
D. Crawler

Solution

  1. Step 1: Identify the role of each component

    The ranking algorithm is responsible for determining the relevance of documents to a query.

  2. Step 2: Eliminate wrong options

    Indexing module organizes data, query parser interprets queries, and crawler collects data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Ranking algorithm determines relevance. [OK]
Hint: Ranking algorithm determines relevance. [OK]
Common Mistakes:
2. In an anomaly detection system using an Autoencoder, what is a likely failure mode if the model is overfitting?
medium
A. The model will detect too many false positives.
B. The model will have a high reconstruction error on the training data.
C. The model will fail to detect any anomalies.
D. The model will require more labeled data to function correctly.

Solution

  1. Step 1: Understand overfitting in Autoencoders.

    Overfitting occurs when the model learns the noise in the training data, leading to poor generalization.

  2. Step 2: Eliminate wrong options.

    The model will fail to detect any anomalies. is incorrect because the model will still detect anomalies, but with poor accuracy. The model will have a high reconstruction error on the training data. is incorrect because overfitting typically results in low reconstruction error on training data. The model will require more labeled data to function correctly. is incorrect because labeled data is not required for Autoencoders.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Overfitting = false positives increase [OK]
Hint: Overfitting = false positives increase [OK]
Common Mistakes:
3. When choosing between a Bi-Encoder and a Cross-Encoder for a semantic search task, which is more suitable for real-time retrieval and why?
medium
A. Bi-Encoder, because it allows pre-computation of document embeddings
B. Cross-Encoder, because it can handle large datasets efficiently
C. Cross-Encoder, because it provides higher accuracy by evaluating pairs
D. Bi-Encoder, because it requires less computational power during retrieval

Solution

  1. Step 1: Compare Bi-Encoder and Cross-Encoder

    Bi-Encoders allow pre-computation of document embeddings, making them suitable for real-time retrieval.

  2. Step 2: Eliminate wrong options

    Cross-Encoders evaluate pairs at query time, which is computationally expensive and not suitable for real-time retrieval.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bi-Encoder = pre-compute embeddings [check]
Hint: Bi-Encoder = pre-compute embeddings [check]
Common Mistakes:
4. As the data scale increases by 10x in an anomaly detection system using Isolation Forest, which component is most likely to become a bottleneck?
hard
A. Model training time due to increased tree depth.
B. Memory usage due to storing more trees.
C. Inference time due to increased number of features.
D. Data preprocessing time due to feature scaling.

Solution

  1. Step 1: Analyze the impact of data scale on Isolation Forest.

    As data increases, the depth of trees in the Isolation Forest can increase, leading to longer training times.

  2. Step 2: Eliminate wrong options.

    Memory usage due to storing more trees. is incorrect because memory usage is more affected by the number of trees, not directly by data scale. Inference time due to increased number of features. is incorrect because inference time is less affected by feature count in Isolation Forests. Data preprocessing time due to feature scaling. is incorrect because feature scaling is a preprocessing step that doesn't scale linearly with data size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    10x data = tree depth increases, training slows [OK]
Hint: 10x data = tree depth increases, training slows [OK]
Common Mistakes:
5. As a recommendation system using neural collaborative filtering scales to 10x more users, which component is most likely to become a bottleneck?
hard
A. Model training time due to increased data volume.
B. Feature extraction pipeline.
C. Real-time recommendation latency.
D. Data storage for user-item interaction history.

Solution

  1. Step 1: Consider the impact of scaling on neural CF.

    Increased data volume significantly affects model training time.

  2. Step 2: Eliminate wrong options.

    Options A and D are less likely bottlenecks as storage and feature extraction can be optimized. Real-time recommendation latency. is affected but not as critically as training time.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Scaling = training time bottleneck [OK]
Hint: Scaling = training time bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard