Bird
Raised Fist0

Diversity and Freshness in Feed - Avoiding Filter Bubbles

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Stale Feed Syndrome
Started last Friday, ongoing for 5 days.

A recommendation system for a social media platform serving 10M users daily, designed to maximize engagement by ensuring diverse and fresh content in user feeds.

Symptom
Recommendation diversity collapsed, AUC dropped from 0.85 to 0.72, and real-time feature latency spiked. A/B test shows offline/online metric disagreement.
📡
Signals
metric
Live AUC dropped significantly on Friday
AUC dropped from 0.85 to 0.72
distribution
Content diversity in recommendations decreased
Diversity index dropped from 0.65 to 0.30
latency
Real-time feature extraction latency increased
P99 latency increased from 50ms to 200ms
feature_importance
Feature 'user_last_active_days' importance changed
Importance dropped from 0.18 to 0.02 after last Thursday's deploy
metric
Offline AUC remains stable
Offline AUC stable at 0.89
🔬
Hypotheses
Feature extraction latency is causing stale data to be used in recommendations.
Supporting Signals
Real-time feature extraction latency increasedLive AUC dropped significantly
Verify: Check timestamps of features used in live predictions to see if they are outdated.
import pandas as pd

# Assuming df is the DataFrame with feature timestamps
df['feature_age'] = pd.Timestamp.now() - df['feature_timestamp']
print(df['feature_age'].describe())
A recent deploy changed feature importance, affecting model predictions.
Supporting Signals
Feature 'user_last_active_days' importance changedLive AUC dropped significantly
Verify: Compare feature importances before and after the deploy to identify changes.
import matplotlib.pyplot as plt

# Assuming feature_importance_df is a DataFrame with feature importances
feature_importance_df.plot(kind='bar')
plt.show()
The model is overfitting to stale training data, causing poor generalization.
Supporting Signals
Offline AUC remains stableLive AUC dropped significantly
Verify: Check the recency of the training data and compare with live data distribution.
import seaborn as sns
import matplotlib.pyplot as plt

# Assuming train_df and live_df are DataFrames with data timestamps
sns.histplot(train_df['timestamp'], label='Train')
sns.histplot(live_df['timestamp'], label='Live', color='red')
plt.legend()
plt.show()
🎯
Root Cause
Feature extraction latency caused stale data to be used in live predictions, reducing recommendation diversity and freshness.
Why It Happened

A recent deployment inadvertently increased the complexity of feature extraction, leading to higher latency and outdated features being used in predictions.

🔧
Fix Strategy
ImmediateRollback to the previous version of the feature extraction pipeline to reduce latency.
Short-termOptimize the current feature extraction process to ensure real-time data freshness.
Long-termImplement a monitoring system for feature extraction latency and data freshness checks before deployment.
🐍
Python Fix
Python
# Example Python code to optimize feature extraction

def optimize_feature_extraction(data):
    # Simplified feature extraction logic
    data['optimized_feature'] = data['raw_feature'].apply(lambda x: x * 0.5)
    return data

# Apply optimization
optimized_data = optimize_feature_extraction(raw_data)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to training data.
Why candidates think this: The stable offline AUC suggests overfitting to training data.
Why wrong: The issue is not overfitting but stale data due to latency, as evidenced by the latency signal.
Feature importance change is the root cause.
Why candidates think this: The drop in feature importance suggests a model issue.
Why wrong: The importance change is a result of stale data, not the cause.
📊
Monitoring to Add
  • Monitor feature extraction latency and alert if P99 exceeds 100ms
  • Check for data freshness in features used for live predictions
  • Track diversity index in recommendations and alert on significant drops
  • Monitor real-time vs offline AUC discrepancies
📋
Process Change

Implement a pre-deployment checklist that includes latency and data freshness tests.

🏗️
Architectural Change

Design the system to use a fallback mechanism that defaults to cached features if real-time extraction exceeds latency thresholds.

💡
Key Insight
Real-time feature latency can lead to stale data usage, collapsing recommendation diversity and freshness.

Practice

(1/5)
1. For an A/B test aimed at increasing user engagement, which metric should the platform optimize for?
easy
A. Conversion Rate
B. Session Duration
C. Click-Through Rate (CTR)
D. Bounce Rate

Solution

  1. Step 1: Define user engagement.

    User engagement is often measured by how long users interact with the platform.

  2. Step 2: Eliminate wrong options.

    CTR and Conversion Rate focus on specific actions, while Bounce Rate measures disengagement.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Engagement = time spent. [OK]
Hint: Engagement = time spent. [OK]
Common Mistakes:
2. For a model registry focused on governance, which metric is most critical to track?
easy
A. Model accuracy
B. Model compliance with regulations
C. Model deployment frequency
D. Model training time

Solution

  1. Step 1: Identify governance focus

    Governance in model registries is about ensuring compliance and adherence to regulations.

  2. Step 2: Eliminate wrong options

    Model accuracy is incorrect as accuracy isn't a governance metric. Model deployment frequency is wrong because deployment frequency isn't about governance. Model training time is incorrect as training time isn't related to governance.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Governance = Compliance focus [OK]
Hint: Governance = Compliance focus [OK]
Common Mistakes:
3. In session-based recommendation systems, which model is most suitable for capturing sequential dependencies in user behavior?
easy
A. Random Forest
B. Recurrent Neural Network (RNN)
C. Support Vector Machine (SVM)
D. K-Nearest Neighbors (KNN)

Solution

  1. Step 1: Identify models that capture sequences

    RNNs are designed to handle sequential data by maintaining a hidden state that captures information from previous inputs.

  2. Step 2: Eliminate non-sequential models

    Random Forest, SVM, and KNN do not inherently capture sequential dependencies.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    RNNs remember sequences. [OK]
Hint: RNNs remember sequences. [OK]
Common Mistakes:
4. In the context of vector search using FAISS, which component is responsible for reducing the dimensionality of vectors before indexing?
easy
A. The preprocessor
B. The index
C. The search algorithm
D. The quantizer

Solution

  1. Step 1: Identify the role of each component.

    The preprocessor is used to reduce dimensionality before indexing.

  2. Step 2: Eliminate wrong options.

    The quantizer is for quantization, the index stores vectors, and the search algorithm performs the search.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Preprocessor reduces dimensions before indexing. [OK]
Hint: Preprocessor reduces dimensions before indexing. [OK]
Common Mistakes:
5. A hate speech detection model is suddenly showing a high false negative rate. What is the most likely cause?
medium
A. The training data is imbalanced with more negative samples.
B. The model's learning rate is too high.
C. The model is overfitting the training data.
D. The feature extraction process is incorrect.

Solution

  1. Step 1: Analyze the symptom

    A high false negative rate often indicates that the model is not sensitive enough to positive samples.

  2. Step 2: Eliminate wrong options

    Overfitting would increase false positives, a high learning rate causes instability, and incorrect feature extraction would affect all predictions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    High false negatives = Imbalanced data [OK]
Hint: High false negatives = Imbalanced data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard