Bird
Raised Fist0

Data Splits - Train, Validation, Test - Why Three Sets

THEORYbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Mysterious Recall Vanishing
Issue started after the latest model deployment last Monday.

A fraud detection model for a financial service, processing 500K transactions daily.

Symptom
Model accuracy remains stable at 0.95, but recall has dropped to 0.
📡
Signals
metric
Recall on the test set
Recall dropped from 0.85 to 0.00 after deployment
metric
Accuracy on the validation set
Accuracy remains stable at 0.95
feature_importance
Feature importance ranking
Feature 'transaction_amount' dropped from rank #1 (0.25) to rank #10 (0.05)
distribution
Distribution of 'transaction_amount' in training vs. test set
Training set mean: 100, Test set mean: 500
log
Deployment logs
No errors or warnings during deployment
🔬
Hypotheses
Data distribution shift between training and test sets
Supporting Signals
distribution
Verify: Check if the distribution of key features differs significantly between training and test sets.
import numpy as np
train_mean = np.mean(train_data['transaction_amount'])
test_mean = np.mean(test_data['transaction_amount'])
print(f'Train mean: {train_mean}, Test mean: {test_mean}')
Feature importance misalignment due to data leakage
Supporting Signals
feature_importance
Verify: Verify if the validation set shares data with the training set, causing overfitting.
train_ids = set(train_data['transaction_id'])
val_ids = set(val_data['transaction_id'])
leakage = train_ids.intersection(val_ids)
print(f'Leaked IDs: {len(leakage)}')
🎯
Root Cause
Data distribution shift between training and test sets
Why It Happened

The test set was not representative of the training data due to a temporal split that did not account for seasonal changes in transaction amounts.

🔧
Fix Strategy
ImmediateRollback to the previous model version.
Short-termRe-evaluate the data split strategy to ensure test set represents the current data distribution.
Long-termImplement a dynamic data splitting strategy that accounts for temporal and seasonal variations.
🐍
Python Fix
Python
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_index, test_index in tscv.split(data):
    train, test = data.iloc[train_index], data.iloc[test_index]
    # Train and evaluate model
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to too many features
Why candidates think this: Feature importance changes suggest overfitting.
Why wrong: The issue is not overfitting but a mismatch in data distribution.
Deployment error causing model malfunction
Why candidates think this: Deployment timing coincides with issue onset.
Why wrong: Deployment logs show no errors; issue is data-related.
📊
Monitoring to Add
  • Monitor recall on live data continuously.
  • Set alerts for significant drops in feature importance.
  • Track distribution changes in key features between training and test sets.
📋
Process Change

Incorporate regular reviews of data splitting strategies to ensure they remain representative.

🏗️
Architectural Change

Adopt a data pipeline that automatically adjusts splits based on detected distribution shifts.

💡
Key Insight
Data splits must be representative of the real-world distribution to ensure model generalization.

Practice

(1/5)
1. Given the following Python code snippet, what will be the output if the model's performance has decayed due to data drift?
import numpy as np
from sklearn.metrics import accuracy_score

y_true = np.array([1, 0, 1, 1, 0])
y_pred = np.array([0, 0, 1, 0, 0])

accuracy = accuracy_score(y_true, y_pred)
print(f'Accuracy: {accuracy:.2f}')
easy
A. Accuracy: 0.40
B. Accuracy: 0.80
C. Accuracy: 0.60
D. Accuracy: 0.20

Solution

  1. Step 1: Calculate accuracy

    Compare y_true and y_pred: only 2 out of 5 predictions are correct.

  2. Step 2: Compute accuracy

    Accuracy = 2/5 = 0.40.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Count correct predictions, divide by total [OK]
Hint: Count correct predictions, divide by total [OK]
Common Mistakes:
2. Given the following Python code snippet, what will be the output if the model is trained with biased data?
from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 0, 0]

accuracy = accuracy_score(y_true, y_pred)
print(accuracy)
easy
A. 0.4
B. 0.8
C. 0.6
D. 0.2

Solution

  1. Step 1: Calculate the accuracy.

    Accuracy is the ratio of correctly predicted instances to the total instances. Here, 3 out of 5 predictions are correct.

  2. Step 2: Compute the accuracy.

    Correct predictions: [0, 1, 1] -> 0.6

  3. Final Answer:

    0.6 -> 0.6

  4. Quick Check:

    Count correct predictions, divide by total. [OK]
Hint: Count correct predictions, divide by total. [OK]
Common Mistakes:
3. A CNN model is overfitting on the training data. Which of the following is a likely cause related to pooling layers?
medium
A. Pooling layers are too aggressive, losing important features
B. Pooling layers are not used, leading to high feature map dimensions
C. Pooling layers are applied after every convolutional layer
D. Pooling layers use a large stride, reducing model complexity

Solution

  1. Step 1: Analyze pooling impact

    Without pooling, feature maps remain large, increasing model complexity and risk of overfitting.

  2. Step 2: Eliminate wrong options

    Pooling layers are too aggressive, losing important features is incorrect because aggressive pooling would reduce overfitting. Pooling layers are applied after every convolutional layer is incorrect as regular pooling helps reduce overfitting. Pooling layers use a large stride, reducing model complexity is incorrect because large strides reduce complexity, not increase it.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No pooling increases overfitting risk. [OK]
Hint: No pooling increases overfitting risk. [OK]
Common Mistakes:
4. You are designing a transformer model for a language translation task. Which normalization technique would you choose and why?
hard
A. Layer Normalization, because it is more suitable for transformer architectures.
B. Batch Normalization, because it is effective for large datasets.
C. Batch Normalization, because it reduces training time.
D. Layer Normalization, because it improves model interpretability.

Solution

  1. Step 1: Consider the architecture of transformers.

    Transformers process sequences and often use self-attention mechanisms, where Layer Normalization is more effective.

  2. Step 2: Evaluate the benefits of Layer Normalization.

    Layer Normalization normalizes across features, which is beneficial for the attention mechanism in transformers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Layer Norm fits transformers' self-attention. [OK]
Hint: Layer Norm fits transformers' self-attention. [OK]
Common Mistakes:
5. You need to apply model interpretability techniques to a real-time fraud detection system. Which approach is most suitable?
hard
A. Use attention mechanisms in the model architecture
B. Use LIME for batch processing of transactions
C. Use SHAP for every transaction in real-time
D. Use a simple decision tree for interpretability

Solution

  1. Step 1: Consider real-time constraints.

    Real-time systems require fast interpretability methods.

  2. Step 2: Evaluate each option.

    SHAP and LIME are computationally expensive for real-time use. Attention mechanisms provide built-in interpretability without significant overhead.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Attention is efficient for real-time interpretability. [OK]
Hint: Attention is efficient for real-time interpretability. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard