Bird
Raised Fist0

Offline vs Online Evaluation - Why They Disagree

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Diagnose Discrepancies Between Offline and Online Evaluation Metrics
40 min · senior
senior

You are part of a team of 5 ML engineers working on a recommendation system for a video streaming platform with 10M DAU. The system uses implicit feedback (watch time, skips) and has been showing promising results offline with a PR-AUC of 0.85. However, the online A/B tests show no significant improvement in user engagement metrics like CTR and watch time. You have 3 months to resolve this discrepancy. The system must maintain a P95 latency SLA of 100ms.

Given Info
  • Offline PR-AUC is 0.85
  • Online metrics (CTR, watch time) show no improvement
  • Implicit feedback data available
  • P95 latency SLA of 100ms
  • 3-month timeline
📋
Guided Steps
1/5
Step 1
Understanding the Discrepancy
Why might the offline PR-AUC not translate to online improvements in CTR and watch time?
💡 Hint
Consider the difference in data distribution and user behavior between offline and online environments.
Key Decision
Identify data distribution shift because offline data may not capture real-time user behavior changes.
Interviewer Expectation
Mid: Recognizes potential data distribution shift. Senior: Explains how user interaction patterns differ in live environments and suggests methods to capture these differences.
⚠️Common Wrong Answer
Assuming offline metrics directly predict online success without considering user behavior changes.
Step 2
Evaluating Feature Relevance
How can you ensure that the features used in the model are relevant to online user behavior?
💡 Hint
Think about feature drift and how features might change over time or differ in importance online.
Key Decision
Implement feature monitoring because feature importance can shift between offline and online settings.
Interviewer Expectation
Mid: Suggests monitoring feature drift. Senior: Proposes specific techniques like online feature importance tracking and real-time feature updates.
⚠️Common Wrong Answer
Ignoring feature drift and assuming static feature importance.
Step 3
Choosing the Right Metrics
What metrics should you use to better align offline and online evaluations?
💡 Hint
Consider metrics that reflect both user engagement and business goals.
Key Decision
Use a combination of engagement metrics and business KPIs because they provide a holistic view of system performance.
Interviewer Expectation
Mid: Proposes using CTR and watch time. Senior: Also includes business impact metrics like revenue per user and explains their relevance.
⚠️Common Wrong Answer
Focusing solely on engagement metrics without considering business impact.
Step 4
Designing an Experiment
How would you design an experiment to test your hypothesis about the discrepancy?
💡 Hint
Think about A/B testing and the importance of controlling variables.
Key Decision
Design a controlled A/B test because it isolates the effect of changes on user behavior.
Interviewer Expectation
Mid: Designs a basic A/B test. Senior: Incorporates control for confounding variables and suggests iterative testing.
⚠️Common Wrong Answer
Running uncontrolled experiments that do not isolate the effect of changes.
Step 5
Iterating on the Model
What changes would you make to the model based on your findings?
💡 Hint
Consider model complexity and the trade-off between accuracy and interpretability.
Key Decision
Adjust model complexity because simpler models may generalize better to online environments.
Interviewer Expectation
Mid: Suggests tuning hyperparameters. Senior: Proposes model architecture changes and justifies them with expected impact on latency and performance.
⚠️Common Wrong Answer
Overfitting the model to offline data without considering online generalization.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Identifies data distribution shift as a potential cause of discrepancy.
  • Proposes monitoring feature drift and adjusting features accordingly.
  • Suggests using a combination of engagement and business metrics.
  • Designs a controlled A/B test to validate hypotheses.
Good to Have - 30%
  • Explains how user behavior differs in online environments.
  • Proposes real-time feature importance tracking.
  • Includes business impact metrics in evaluation.
  • Incorporates control for confounding variables in experiments.
Bonus - 10%
  • Suggests iterative testing with feedback loops.
  • Proposes model architecture changes with latency considerations.
  • Quantifies expected impact of changes on online metrics.
✅
Model Answer
▼ Reveal

The ideal solution identifies the data distribution shift as a key reason for the discrepancy between offline and online metrics. It involves monitoring feature drift and adjusting features to align with real-time user behavior. The solution uses a combination of engagement and business metrics to evaluate performance and designs controlled A/B tests to validate hypotheses. Iterative testing and model adjustments are made to ensure the model generalizes well to online environments.

Key Decisions
  • Identify data distribution shift because offline data may not capture real-time user behavior changes.
  • Implement feature monitoring because feature importance can shift between offline and online settings.
  • Use a combination of engagement metrics and business KPIs because they provide a holistic view of system performance.
  • Design a controlled A/B test because it isolates the effect of changes on user behavior.
  • Adjust model complexity because simpler models may generalize better to online environments.
Anti-patterns
Assuming offline metrics directly predict online success.
Ignoring feature drift and assuming static feature importance.
Focusing solely on engagement metrics without considering business impact.
Running uncontrolled experiments that do not isolate the effect of changes.

Practice

(1/5)
1. Given the following Python code, what will be the output? 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(len(X_train), len(X_test))
easy
A. 6 4
B. 8 2
C. 7 3
D. 5 5

Solution

  1. Step 1: Calculate the split sizes

    With a test_size of 0.2, 20% of 10 samples go to the test set, i.e., 2 samples.

  2. Step 2: Determine the training set size

    The remaining 80% (8 samples) go to the training set.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    80% train, 20% test split. [OK]
Hint: 80% train, 20% test split. [OK]
Common Mistakes:
2. Given the following Python code snippet, what will be the output? 
import numpy as np

data = np.array([1, 2, 3, 4, 5])

# Simulate offline data processing
processed_data = data * 2

print(processed_data)
easy
A. [1, 2, 3, 4, 5]
B. [2, 4, 6, 8, 10]
C. [1, 4, 9, 16, 25]
D. Error: Data processing requires real-time input

Solution

  1. Step 1: Understand the code operation

    The code multiplies each element of the array by 2.

  2. Step 2: Calculate the expected output

    Each element in the array [1, 2, 3, 4, 5] is multiplied by 2, resulting in [2, 4, 6, 8, 10].

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Array * 2 doubles each element. [OK]
Hint: Array * 2 doubles each element. [OK]
Common Mistakes:
3. A model trained on standardized data is underperforming. What could be a potential issue?
medium
A. The model is sensitive to the mean of the features.
B. The standard deviation of features was not considered.
C. The data was standardized but not normalized, causing issues.
D. The data was not normalized, affecting model performance.

Solution

  1. Step 1: Analyze the impact of standardization on model performance.

    Standardization centers data around zero, which can affect models sensitive to feature means.

  2. Step 2: Identify the potential issue.

    Some models may perform poorly if they rely on the original mean of the data.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Model sensitive to mean? Standardization affects it. [OK]
Hint: Model sensitive to mean? Standardization affects it. [OK]
Common Mistakes:
4. When choosing between Least Squares and Gradient Descent for linear regression, which scenario makes Gradient Descent more suitable?
medium
A. When the dataset is small and fits in memory
B. When the dataset has a lot of noise
C. When the dataset is very large and doesn't fit in memory
D. When the model needs to be interpretable

Solution

  1. Step 1: Identify the constraint

    Gradient Descent is more suitable for large datasets that don't fit in memory because it can process data in batches.

  2. Step 2: Eliminate wrong options

    Least Squares is efficient for small datasets (A), interpretability is not a deciding factor (C), and noise handling depends on regularization, not the optimization method (D).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Gradient Descent handles large datasets in batches. [OK]
Hint: Gradient Descent handles large datasets in batches. [OK]
Common Mistakes:
5. How would you adapt K-Means to handle a dataset with categorical features?
hard
A. Use one-hot encoding and apply standard K-Means
B. Use K-Modes clustering instead
C. Convert categorical features to numerical using label encoding
D. Ignore categorical features and cluster on numerical features only

Solution

  1. Step 1: Understand K-Means limitations

    K-Means is not suitable for categorical data as it relies on Euclidean distance.

  2. Step 2: Choose the right adaptation

    K-Modes is designed for clustering categorical data, making it the best choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    K-Modes for categorical data [OK]
Hint: K-Modes for categorical data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard