Bird
Raised Fist0

Data Leakage - The Silent Killer of ML Models

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/4
Data Collection

Raw data is gathered, including both training and test sets.

🎯 Ensure separation of training and test data from the start.
data = pd.read_csv('dataset.csv')
train_data, test_data = train_test_split(data, test_size=0.2)
Feature Engineering

Features are created using both training and test data, risking leakage.

🎯 Feature engineering should be done only on training data.
train_features = feature_engineering(train_data)
# Avoid using test_data here
Model Training

Model is trained on features, unaware of potential leakage.

🎯 Leakage can lead to overly optimistic model performance.
model.fit(train_features, train_labels)
Model Evaluation

Model is evaluated on test data, results may be skewed by leakage.

🎯 Evaluate models on unseen data to ensure true performance.
predictions = model.predict(test_features)
accuracy = accuracy_score(test_labels, predictions)
📊
Diagram
Diagram
┌──────────────────────────────┐
│      Data Collection         │
└────────────┬─────────────────┘
             │
             ▼
┌──────────────────────────────┐
│      Feature Engineering     │
│  (Potential Leakage Point)   │
└────────────┬─────────────────┘
             │
             ▼
┌──────────────────────────────┐
│      Model Training          │
│  (Leakage Affects Accuracy)  │
└────────────┬─────────────────┘
             │
             ▼
┌──────────────────────────────┐
│      Model Evaluation        │
│  (Leakage Skews Results)     │
└──────────────────────────────┘
Labels
  • Data Collection: Initial data gathering phase
  • Feature Engineering: Transformation of raw data, potential leakage point
  • Model Training: Learning phase, leakage affects accuracy
  • Model Evaluation: Testing phase, leakage skews results
🔑
Key Moments
▲
Q: Why is feature engineering a critical point for data leakage?
A: Feature engineering can inadvertently use information from the test set, leading to leakage.
Q: How does data leakage affect model evaluation?
A: Leakage can cause the model to perform well on test data due to information overlap, not true generalization.
Q: What is a key strategy to prevent data leakage during model training?
A: Ensure that feature engineering and model training are conducted only on the training dataset.
Trace Summary

The diagram illustrates the flow of data through an ML pipeline, emphasizing the critical points where data leakage can occur. Data leakage happens when information from the test set is used during training, often during feature engineering, leading to overly optimistic model performance. This can skew model evaluation results, making the model appear more accurate than it truly is. To prevent leakage, it's crucial to separate training and test data from the beginning and ensure that feature engineering is only applied to the training data. Understanding and mitigating data leakage is essential for building robust and reliable ML models.

Primary Focus

✓ Understanding and preventing data leakage in ML models

Practice

(1/5)
1. Given the following Python code, what is the output when evaluating the calibration of a model using Brier Score?
from sklearn.metrics import brier_score_loss
true_labels = [0, 1, 1, 0]
predicted_probs = [0.1, 0.9, 0.8, 0.4]
score = brier_score_loss(true_labels, predicted_probs)
print(score)
easy
A. 0.35
B. 0.15
C. 0.05
D. 0.25

Solution

  1. Step 1: Understand Brier Score calculation.

    Brier Score is the mean squared difference between predicted probabilities and actual outcomes.

  2. Step 2: Calculate the score.

    For each pair: (0.1-0)^2, (0.9-1)^2, (0.8-1)^2, (0.4-0)^2. Mean = (0.01 + 0.01 + 0.04 + 0.16) / 4 = 0.055.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Brier Score = mean squared error of probabilities. [OK]
Hint: Brier Score = mean squared error of probabilities. [OK]
Common Mistakes:
2. In the context of distributional shift, which type of drift occurs when the statistical properties of the input features change over time?
easy
A. Label drift
B. Concept drift
C. Covariate drift
D. Feature drift

Solution

  1. Step 1: Identify the type of drift.

    Covariate drift refers to changes in the distribution of input features.

  2. Step 2: Eliminate wrong options.

    Label drift involves changes in the distribution of the output labels. Concept drift involves changes in the relationship between inputs and outputs. Feature drift is not a standard term.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Covariate drift = input feature change [OK]
Hint: Covariate drift = input feature change [OK]
Common Mistakes:
3. In a dataset with highly imbalanced classes, which sampling strategy is most appropriate to ensure each class is represented proportionally in the sample?
easy
A. Stratified Sampling
B. Weighted Sampling
C. Reservoir Sampling
D. Random Sampling

Solution

  1. Step 1: Identify the need for proportional representation.

    In imbalanced datasets, it's crucial to maintain class proportions to avoid bias.

  2. Step 2: Eliminate inappropriate strategies.

    Random Sampling (A) doesn't guarantee class proportions. Reservoir Sampling (C) is for streaming data, not class balance. Weighted Sampling (D) adjusts for importance, not proportions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stratified = Proportional representation [OK]
Hint: Stratified = Proportional representation [OK]
Common Mistakes:
4. A model registered in MLflow Registry is not showing the expected performance in production. What is the most likely cause?
medium
A. The model version was not updated in the registry.
B. The model was not packaged correctly in MLflow Models.
C. The model's hyperparameters were not logged correctly.
D. The model was trained with a different dataset than intended.

Solution

  1. Step 1: Analyze potential causes of performance issues

    Data discrepancies are a common cause of unexpected model performance.

  2. Step 2: Eliminate less likely causes

    Incorrect dataset usage is more likely than versioning or packaging issues.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Check dataset consistency [OK]
Hint: Check dataset consistency [OK]
Common Mistakes:
5. When should you prefer using LightGBM over XGBoost for a gradient boosting task?
medium
A. When you have a very large dataset with high dimensionality.
B. When you need better accuracy on small datasets.
C. When interpretability of the model is the primary concern.
D. When you require a model that handles categorical features natively.

Solution

  1. Step 1: Identify LightGBM's strengths

    LightGBM is optimized for speed and performance on large datasets with high dimensionality.

  2. Step 2: Eliminate wrong options

    When you need better accuracy on small datasets. is incorrect because XGBoost often performs better on small datasets. When interpretability of the model is the primary concern. is incorrect because neither LightGBM nor XGBoost is particularly interpretable. When you require a model that handles categorical features natively. is incorrect because both can handle categorical features, but LightGBM is faster on large datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    LightGBM = Large, high-dimensional data [OK]
Hint: LightGBM = Large, high-dimensional data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard