Bird
Raised Fist0

ROC-AUC vs PR-AUC - Imbalanced Classes, Why AUC Misleads

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Imbalance Trap
Started immediately after the last feature deployment, ongoing for 3 days.

Fraud detection model for a financial services company, processing 500K transactions per day.

Symptom
Model accuracy is 98%, but recall is 0. AUC dropped from 0.85 to 0.65 after adding a new feature.
📡
Signals
metric
PR-AUC dropped significantly after the last deployment.
PR-AUC dropped from 0.75 to 0.30.
metric
ROC-AUC remained relatively stable despite the drop in PR-AUC.
ROC-AUC is at 0.85, similar to pre-deployment.
feature_importance
New feature 'transaction_time_of_day' has high importance.
Feature importance for 'transaction_time_of_day' is 0.25, ranked #1.
distribution
Class distribution is highly imbalanced.
Fraudulent transactions make up only 0.5% of the dataset.
log
Error logs show no anomalies or exceptions.
No errors or warnings in logs post-deployment.
🔬
Hypotheses
The new feature 'transaction_time_of_day' is causing the model to overfit to non-fraudulent transactions.
Supporting Signals
feature_importancedistribution
Verify: Check the correlation between 'transaction_time_of_day' and the target variable.
df[['transaction_time_of_day', 'target']].corr()
The model's evaluation metric is not suitable for imbalanced classes, leading to misleading ROC-AUC.
Supporting Signals
metricdistribution
Verify: Compare ROC-AUC and PR-AUC on a balanced validation set.
from sklearn.metrics import roc_auc_score, average_precision_score; roc_auc_score(y_true_balanced, y_pred_balanced), average_precision_score(y_true_balanced, y_pred_balanced)
🎯
Root Cause
The model's reliance on ROC-AUC as a primary metric masked the poor performance on the minority class.
Why It Happened

The addition of a new feature skewed the model's focus, and the evaluation metric was not sensitive to class imbalance.

🔧
Fix Strategy
ImmediateRollback to the previous model version to prevent further misclassification.
Short-termRe-evaluate the model using PR-AUC and adjust the threshold for fraud detection to improve recall.
Long-termIncorporate class-weighted loss functions and use PR-AUC as a primary evaluation metric.
🐍
Python Fix
Python
from sklearn.metrics import precision_recall_curve; precision, recall, thresholds = precision_recall_curve(y_true, y_scores); optimal_idx = np.argmax(precision * recall); optimal_threshold = thresholds[optimal_idx]
✗
Common Wrong Diagnoses
▼ Show
The new feature 'transaction_time_of_day' is causing data leakage.
Why candidates think this: High feature importance suggests it might be leaking target information.
Why wrong: No evidence of data leakage in logs or feature correlation analysis.
The model is overfitting due to the new feature.
Why candidates think this: High feature importance and drop in AUC suggest overfitting.
Why wrong: Overfitting would typically affect both ROC-AUC and PR-AUC, not just PR-AUC.
📊
Monitoring to Add
  • Monitor PR-AUC alongside ROC-AUC for all deployments.
  • Set up alerts for significant drops in recall.
  • Track feature importance changes post-deployment.
📋
Process Change

Include PR-AUC in the model evaluation criteria during development and testing.

🏗️
Architectural Change

Implement a balanced validation set for all model evaluations to ensure metrics reflect true performance.

💡
Key Insight
ROC-AUC can be misleading in imbalanced datasets; PR-AUC provides a clearer picture of model performance on minority classes.

Practice

(1/5)
1. In a binary classification problem where false positives are costly, which metric should be prioritized?
easy
A. Precision
B. F1 Score
C. Recall
D. Accuracy

Solution

  1. Step 1: Identify the cost concern.

    False positives are costly, so we need to minimize them.

  2. Step 2: Choose the metric that minimizes false positives.

    Precision is the metric that focuses on minimizing false positives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Precision minimizes false positives. [OK]
Hint: Precision minimizes false positives. [OK]
Common Mistakes:
2. 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 1, 0, 1])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)

print(X_train.shape, X_test.shape)
What is the output of this code snippet?
easy
A. (2, 2) (2, 1)
B. (3, 2) (1, 2)
C. (2, 2) (2, 2)
D. (1, 2) (3, 2)

Solution

  1. Step 1: Understand train_test_split behavior.

    With test_size=0.5, the data is split equally into train and test sets.

  2. Step 2: Calculate expected shapes.

    Original data has 4 samples, so both train and test sets will have 2 samples each.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Equal split with test_size=0.5. [OK]
Hint: Equal split with test_size=0.5. [OK]
Common Mistakes:
3. In the context of evaluating a recommendation system, which of the following best describes why offline evaluation metrics might disagree with online performance?
easy
A. Offline metrics are influenced by server latency.
B. Offline metrics are always more accurate than online metrics.
C. Online metrics are biased due to data leakage.
D. Offline metrics may not capture user interaction dynamics.

Solution

  1. Step 1: Understand the difference between offline and online evaluation.

    Offline evaluation uses historical data and predefined metrics, while online evaluation captures real-time user interactions.

  2. Step 2: Eliminate wrong options.

    Offline metrics are always more accurate than online metrics. is incorrect because offline metrics are not inherently more accurate. Online metrics are biased due to data leakage. is incorrect because online metrics are not biased due to data leakage; they reflect real-world interactions. Offline metrics are influenced by server latency. is incorrect because server latency affects online performance, not offline metrics.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Offline misses real user dynamics. [OK]
Hint: Offline misses real user dynamics. [OK]
Common Mistakes:
4. 
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 1, 0, 1])

model = DecisionTreeClassifier()
scores = cross_val_score(model, X, y, cv=2)
print(scores)
What does this code output?
easy
A. An array of two accuracy scores, each representing a fold.
B. An error because the dataset is too small for cross-validation.
C. A single accuracy score for the entire dataset.
D. An array of four accuracy scores, one for each data point.

Solution

  1. Step 1: Understand cross-validation output

    Cross-validation with cv=2 splits the data into two folds, returning an array of two scores.

  2. Step 2: Eliminate wrong options

    An error because the dataset is too small for cross-validation. is incorrect because the code runs without error. A single accuracy score for the entire dataset. is incorrect because cross-validation returns multiple scores. An array of four accuracy scores, one for each data point. is incorrect because the number of scores equals the number of folds, not data points.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cross-validation returns scores per fold. [OK]
Hint: Cross-validation returns scores per fold. [OK]
Common Mistakes:
5. After applying PCA to your dataset, you notice that the variance explained by the components is much lower than expected. What is the most likely cause?
medium
A. The number of components is too high.
B. The dataset has too many missing values.
C. The features are not standardized.
D. The dataset is too small.

Solution

  1. Step 1: Identify the role of feature scaling in PCA

    PCA is sensitive to the scale of the data.

  2. Step 2: Evaluate each option

    Missing values would cause errors, not low variance. Too many components would explain more variance, not less. A small dataset affects generalization, not variance explained.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Standardize features before PCA [OK]
Hint: Standardize features before PCA [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard