Bird
Raised Fist0

Online vs Offline Data Collection - Tradeoffs at Scale

THEORYadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The AUC Anomaly
Started after a new feature deployment last Friday.

A recommendation system for a streaming service, serving 5M users daily, using a hybrid online-offline data collection strategy.

Symptom
Model achieves 0.92 AUC offline but drops to 0.75 AUC on live traffic.
📡
Signals
metric
Offline AUC from validation set
0.92 consistently over the last month
metric
Live AUC on production traffic
Dropped from 0.90 to 0.75 after last Friday's deploy
feature_importance
Feature importance ranking
Feature 'recent_watch_time' dropped from rank #3 (0.15) to rank #10 (0.05)
distribution
Distribution of 'recent_watch_time' feature
Shifted significantly post-deploy, mean increased by 30%
log
Logs from data collection pipeline
Increased volume of 'recent_watch_time' data post-deploy
🔬
Hypotheses
Data distribution shift in 'recent_watch_time' feature
Supporting Signals
Distribution of 'recent_watch_time' featureFeature importance ranking
Verify: Check if the distribution shift correlates with AUC drop
import matplotlib.pyplot as plt
plt.hist(df['recent_watch_time'], bins=50)
plt.title('Recent Watch Time Distribution')
plt.show()
Feature 'recent_watch_time' is being incorrectly logged
Supporting Signals
Logs from data collection pipelineDistribution of 'recent_watch_time' feature
Verify: Audit logging mechanism for 'recent_watch_time'
print(df['recent_watch_time'].describe())
🎯
Root Cause
The 'recent_watch_time' feature was incorrectly logged due to a bug in the data collection script.
Why It Happened

A recent deployment changed the logging format, causing incorrect data to be collected.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the data collection script.
Short-termCorrect the logging format in the data collection script and redeploy.
Long-termImplement a validation step in the data pipeline to catch format changes.
🐍
Python Fix
Python
def correct_logging_format(data):
    # Correct the logging format
    return data.apply(lambda x: correct_format(x))

corrected_data = correct_logging_format(df['recent_watch_time'])
✗
Common Wrong Diagnoses
▼ Show
Model overfitting to offline data
Why candidates think this: The offline AUC is high, suggesting a potential overfit.
Why wrong: The issue is not with model complexity but with data inconsistency between offline and online environments.
Feature engineering error
Why candidates think this: Feature importance changed significantly.
Why wrong: The change in feature importance is due to incorrect data logging, not feature engineering.
📊
Monitoring to Add
  • Real-time monitoring of feature distributions in production
  • Alerts for significant shifts in feature importance
  • Automated checks for data format consistency post-deploy
📋
Process Change

Introduce a mandatory review process for changes in data logging scripts.

🏗️
Architectural Change

Adopt a schema registry to enforce data format consistency across deployments.

💡
Key Insight
Data collection consistency is crucial; even small logging changes can lead to significant model performance issues.

Practice

(1/5)
1. 
from sklearn.tree import DecisionTreeClassifier
import numpy as np

X = np.array([[0, 0], [1, 1], [1, 0], [0, 1]])
y = np.array([0, 1, 1, 0])

clf = DecisionTreeClassifier(criterion='gini')
clf.fit(X, y)

print(clf.predict([[0, 0], [1, 1]]))
What is the output of the above code?
easy
A. [0, 0]
B. [1, 0]
C. [0, 1]
D. [1, 1]

Solution

  1. Step 1: Understand the training data.

    The data is perfectly separable, and the decision tree will learn to classify based on the given labels.

  2. Step 2: Predict using the trained model.

    For input [0, 0], the label is 0, and for [1, 1], the label is 1.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check training labels for predictions. [OK]
Hint: Check training labels for predictions. [OK]
Common Mistakes:
2. A neural network model is overfitting on the training data. What is the most likely cause?
medium
A. The learning rate is too high.
B. The model is too complex for the amount of training data.
C. The model has too few parameters.
D. The training data is too diverse.

Solution

  1. Step 1: Identify symptoms of overfitting

    Overfitting occurs when a model performs well on training data but poorly on unseen data.

  2. Step 2: Determine the cause

    A complex model with too many parameters relative to the data size is a common cause of overfitting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Complex models overfit small data. [OK]
Hint: Complex models overfit small data. [OK]
Common Mistakes:
3. When should you prefer using Gini Impurity over Entropy in decision trees?
medium
A. When computational efficiency is a priority.
B. When you need a more interpretable model.
C. When the dataset is highly imbalanced.
D. When you need to handle continuous features.

Solution

  1. Step 1: Compare computational efficiency.

    Gini Impurity is computationally less intensive than Entropy.

  2. Step 2: Eliminate wrong options.

    Interpretability is not affected by the choice, imbalance handling is not specific to Gini, and both can handle continuous features.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Gini is faster than Entropy. [OK]
Hint: Gini is faster than Entropy. [OK]
Common Mistakes:
4. A Graph Neural Network model is underperforming on a node classification task. The model's architecture and hyperparameters seem appropriate. What could be a potential cause of this issue?
medium
A. The model has too many layers, leading to overfitting.
B. The learning rate is too high, causing unstable training.
C. The batch size is too small, leading to poor generalization.
D. The graph data has isolated nodes, affecting message passing.

Solution

  1. Step 1: Analyze the graph structure

    Isolated nodes can disrupt message passing, leading to poor performance.

  2. Step 2: Eliminate other causes

    Overfitting, learning rate, and batch size issues would manifest differently.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Check for isolated nodes disrupting message passing. [check]
Hint: Check for isolated nodes disrupting message passing. [check]
Common Mistakes:
5. You are tasked with designing a recommendation system for a high-dimensional dataset. Which approach would best handle the curse of dimensionality while maintaining performance?
hard
A. Use a rule-based system to manually curate recommendations.
B. Use collaborative filtering with matrix factorization.
C. Apply a deep learning model with many hidden layers.
D. Implement a simple content-based filtering system.

Solution

  1. Step 1: Identify the challenge of high dimensions.

    Matrix factorization reduces dimensionality effectively.

  2. Step 2: Eliminate wrong options.

    Implement a simple content-based filtering system. doesn't reduce dimensions. Apply a deep learning model with many hidden layers. may overfit due to high dimensions. Use a rule-based system to manually curate recommendations. is not scalable for large datasets.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Matrix factorization reduces dimensions, aids recommendations. [OK]
Hint: Matrix factorization reduces dimensions, aids recommendations. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard