Bird
Raised Fist0

Feed Ranking System - Twitter, Instagram, Facebook at Scale

SYSTEM-DESIGNhardInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Weekend Model Collapse
Started Friday evening, ongoing for 3 days.

A feed ranking system for a social media platform serving 500M daily active users, optimizing for user engagement and content diversity.

Symptom
AUC dropped from 0.85 to 0.72 over the past weekend, recommendation diversity collapsed, and real-time feature latency spiked.
📡
Signals
metric
Live AUC observed in production dashboards
Dropped from 0.85 to 0.72 over the weekend
distribution
Distribution of 'user_engagement_score' feature
Shifted significantly with a spike in zero values post-deploy
latency
Real-time feature computation latency
Increased from P99 < 50ms to P99 = 200ms
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 after last Thursday's deploy
log
Error logs from feature computation service
Increased frequency of 'timeout' errors since Friday
🔬
Hypotheses
Feature computation service is experiencing increased latency, affecting real-time features.
Supporting Signals
latencylog
Verify: Check service logs for timeout errors and latency metrics.
import pandas as pd
logs = pd.read_csv('service_logs.csv')
timeouts = logs[logs['error'] == 'timeout']
print(timeouts['timestamp'].value_counts())
A recent deploy introduced a bug in feature extraction, causing distribution shift.
Supporting Signals
distributionfeature_importance
Verify: Compare feature distributions before and after the deploy.
import matplotlib.pyplot as plt
before = pd.read_csv('features_before.csv')
after = pd.read_csv('features_after.csv')
before['user_engagement_score'].hist(alpha=0.5, label='Before')
after['user_engagement_score'].hist(alpha=0.5, label='After')
plt.legend()
plt.show()
🎯
Root Cause
A bug in the feature extraction code deployed on Thursday caused incorrect feature values, leading to degraded model performance.
Why It Happened

A recent code change introduced a bug in the feature extraction logic, which was not caught in testing due to insufficient coverage of edge cases.

🔧
Fix Strategy
ImmediateRollback to the last known good model and feature extraction code.
Short-termFix the bug in the feature extraction code and redeploy after thorough testing.
Long-termImplement a more robust testing framework for feature extraction, including edge case coverage.
🐍
Python Fix
Python
# Example fix for feature extraction bug
import pandas as pd
def correct_feature_extraction(data):
    # Correct the logic here
    data['user_engagement_score'] = data['activity_count'] / (data['days_active'] + 1)
    return data
# Apply the fix
corrected_data = correct_feature_extraction(raw_data)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to insufficient training data.
Why candidates think this: The AUC drop suggests a model performance issue.
Why wrong: The stable validation AUC indicates the model itself is not overfitting.
The recommendation algorithm is not considering diversity constraints.
Why candidates think this: The collapse in recommendation diversity suggests a constraint issue.
Why wrong: The diversity collapse is a result of incorrect feature values, not algorithmic constraints.
📊
Monitoring to Add
  • Real-time feature distribution monitoring with alerts for significant shifts
  • Latency monitoring for feature computation services with alerting
  • A/B test monitoring for offline/online metric discrepancies
  • Feature importance drift detection
📋
Process Change

Introduce a mandatory code review process for all feature extraction changes with a focus on edge case testing.

🏗️
Architectural Change

Adopt a feature store architecture to decouple feature computation from model serving, allowing for independent validation and rollback.

💡
Key Insight
Feature extraction bugs can cause significant model performance degradation, often masked by stable offline metrics.

Practice

(1/5)
1. For a fine-tuning pipeline using RLHF, which metric is most appropriate to evaluate the alignment of model outputs with human preferences?
easy
A. Precision
B. Reward Score
C. Accuracy
D. F1 Score

Solution

  1. Step 1: Understand the purpose of RLHF

    RLHF aims to align model outputs with human preferences, which is best evaluated by a reward score reflecting human feedback.

  2. Step 2: Eliminate wrong options

    A: Accuracy is too generic. B: Precision is for binary classification. D: F1 Score is for balance between precision and recall, not alignment.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Reward Score reflects human feedback. [OK]
Hint: Reward Score reflects human feedback. [OK]
Common Mistakes:
2. When choosing between behavioral signals and device fingerprinting for bot detection, which is more suitable for detecting sophisticated bots and why?
medium
A. Behavioral signals, because they are harder for bots to mimic.
B. Device fingerprinting, because it provides a unique identifier.
C. Behavioral signals, because they are easier to implement.
D. Device fingerprinting, because it is less prone to false positives.

Solution

  1. Step 1: Consider sophistication of bots.

    Sophisticated bots can mimic device fingerprints but struggle with human-like behavior.

  2. Step 2: Eliminate wrong options.

    Device fingerprinting is easier to spoof; implementation ease is irrelevant to detection capability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sophisticated bots struggle with human behavior [OK]
Hint: Sophisticated bots struggle with human behavior [OK]
Common Mistakes:
3. In a two-stage retrieval system, when should you prioritize a simpler candidate generation model over a complex one?
medium
A. When the ranking stage is computationally expensive and needs more resources.
B. When precision is more important than recall in the candidate generation stage.
C. When the dataset is small and computation resources are abundant.
D. When the ranking model is not capable of handling large candidate sets.

Solution

  1. Step 1: Consider resource allocation

    If the ranking stage is computationally expensive, a simpler candidate generation model can save resources for the ranking stage.

  2. Step 2: Eliminate wrong options

    When the dataset is small and computation resources are abundant. is incorrect because a small dataset doesn't necessitate a simpler model. When precision is more important than recall in the candidate generation stage. is incorrect because recall is typically prioritized in candidate generation. When the ranking model is not capable of handling large candidate sets. is incorrect because the ranking model should handle the candidate set size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Save resources for ranking if it's costly. [OK]
Hint: Save resources for ranking if it's costly. [OK]
Common Mistakes:
4. As the user base scales 10x, which component of a DLRM-based CTR prediction system is most likely to become a bottleneck?
hard
A. Data preprocessing pipeline
B. Training data storage capacity
C. Model inference latency
D. Embedding table size

Solution

  1. Step 1: Consider scaling effects on components.

    Embedding tables grow with user and feature diversity.

  2. Step 2: Eliminate wrong options.

    Preprocessing (A) and inference (B) scale with compute resources. Storage (D) is less critical than real-time constraints.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Embedding tables grow with scale. [OK]
Hint: Embedding tables grow with scale. [OK]
Common Mistakes:
5. As the scale of data increases in a semantic search system using Bi-Encoders, which component is likely to fail first and why?
hard
A. Network bandwidth, due to increased data transfer
B. Document storage, due to increased storage requirements
C. Query processing, due to increased latency
D. Embedding computation, due to increased computational load

Solution

  1. Step 1: Analyze the impact of data scale on components

    As data scales, the computational load for embedding computation increases significantly.

  2. Step 2: Eliminate wrong options

    Query processing and network bandwidth are affected but not as critically as embedding computation. Document storage is a concern but not a failure point.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scale = embedding load [check]
Hint: Scale = embedding load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard