Bird
Raised Fist0

Data Leakage - The Silent Killer of ML Models

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🎯
Diagnosing and Mitigating Data Leakage in a Financial Fraud Detection Model
40 min · senior
senior

You are part of a team of 5 ML engineers working for a financial institution. Your task is to improve a fraud detection model that processes 1 million transactions per day. The model currently shows high accuracy during offline evaluation but performs poorly in production. You have access to transaction logs, customer profiles, and historical fraud labels. The model must make predictions in under 100ms.

Given Info
  • The model is trained on a rolling window of the last 6 months of data.
  • Feature engineering includes transaction amount, location, and customer behavior patterns.
  • The model's offline AUC is 0.95, but online AUC drops to 0.70.
  • The team suspects data leakage but hasn't identified the source.
  • The model must comply with a 100ms prediction latency requirement.
📋
Guided Steps
1/5
Step 1
Identifying Symptoms of Data Leakage
What are the key indicators that suggest data leakage might be occurring in this scenario?
💡 Hint
Consider discrepancies between offline and online performance metrics.
Key Decision
Focus on performance discrepancies because they often signal leakage.
Interviewer Expectation
Mid: Identifies performance drop as a symptom. Senior: Also considers feature leakage and temporal leakage as potential causes.
⚠️Common Wrong Answer
Assuming the model is overfitting without considering data leakage.
Step 2
Analyzing Feature Engineering
How might the current feature engineering process contribute to data leakage?
💡 Hint
Think about how features are derived and their temporal alignment with labels.
Key Decision
Review feature derivation because misalignment can cause leakage.
Interviewer Expectation
Mid: Identifies potential leakage from future data. Senior: Proposes specific checks for temporal alignment and feature leakage.
⚠️Common Wrong Answer
Ignoring the possibility of future data being used in feature creation.
Step 3
Evaluating the Data Pipeline
What changes can be made to the data pipeline to prevent data leakage?
💡 Hint
Consider how data is split and processed over time.
Key Decision
Implement strict temporal splits because they prevent leakage from future data.
Interviewer Expectation
Mid: Suggests temporal validation. Senior: Designs a robust pipeline with clear data boundaries and checks.
⚠️Common Wrong Answer
Relying solely on random splits without temporal consideration.
Step 4
Balancing Latency and Model Complexity
How can you ensure the redesigned model meets the latency requirement?
💡 Hint
Consider the trade-off between model complexity and prediction speed.
Key Decision
Optimize model complexity because it affects latency.
Interviewer Expectation
Mid: Proposes simpler models. Senior: Quantifies latency impact and suggests specific optimizations.
⚠️Common Wrong Answer
Choosing a complex model without considering latency constraints.
Step 5
Validating the Redesigned System
How will you validate that the redesigned system effectively mitigates data leakage?
💡 Hint
Think about both offline and online validation strategies.
Key Decision
Use both offline and online validation because they provide comprehensive insights.
Interviewer Expectation
Mid: Proposes offline validation. Senior: Integrates online A/B testing and monitoring.
⚠️Common Wrong Answer
Relying only on offline validation without real-world testing.
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Identifies performance discrepancy as a key symptom of data leakage.
  • Proposes temporal validation to prevent leakage from future data.
  • Suggests changes to the data pipeline to enforce strict data boundaries.
  • Considers latency constraints when redesigning the model.
Good to Have - 30%
  • Analyzes feature engineering for potential leakage sources.
  • Designs a robust data pipeline with clear checks for leakage.
  • Quantifies the impact of model complexity on latency.
  • Integrates online validation strategies like A/B testing.
Bonus - 10%
  • Proposes specific optimizations to balance model complexity and latency.
  • Designs a monitoring system to detect leakage in production.
  • Explains the trade-offs between different validation strategies.
✅
Model Answer
▼ Reveal

The ideal solution identifies data leakage as the cause of performance discrepancies and proposes a redesign of the data pipeline to enforce strict temporal boundaries. It balances model complexity with latency requirements and validates the solution through both offline and online testing.

Key Decisions
  • Identifying performance discrepancies as a symptom of data leakage.
  • Implementing temporal validation to prevent future data leakage.
  • Redesigning the data pipeline to enforce strict data boundaries.
  • Balancing model complexity with latency requirements.
  • Validating the redesigned system through both offline and online testing.
Anti-patterns
Assuming overfitting without considering data leakage.
Ignoring temporal alignment in feature engineering.
Relying solely on random data splits for validation.
Choosing complex models without latency consideration.

Practice

(1/5)
1. What will be the output shape of the following code snippet using K-Fold cross-validation with 5 splits on a dataset of 100 samples?
from sklearn.model_selection import KFold
kf = KFold(n_splits=5)
for train_index, test_index in kf.split(range(100)):
    print(len(test_index))
easy
A. 20, 20, 20, 20, 20
B. 25, 25, 25, 25
C. 20, 20, 20, 20, 20, 20
D. 20, 20, 20, 20

Solution

  1. Step 1: Calculate the number of samples per fold.

    100 samples divided by 5 splits results in 20 samples per fold.

  2. Step 2: Eliminate wrong options.

    25 samples per fold (B) is incorrect, 6 outputs (C) is incorrect, 4 outputs (D) is incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    100 samples, 5 splits = 20 per fold. [OK]
Hint: 100 samples, 5 splits = 20 per fold. [OK]
Common Mistakes:
2. What will be the output of the following Python code snippet using NumPy for a simple linear regression prediction?
import numpy as np
X = np.array([[1, 2], [3, 4]])
weights = np.array([0.5, 0.5])
predictions = np.dot(X, weights)
print(predictions)
easy
A. [1.0, 2.0]
B. [2.5, 3.5]
C. [1.5, 3.5]
D. [2.0, 4.0]

Solution

  1. Step 1: Calculate the dot product

    The dot product of X and weights is calculated as follows: [1*0.5 + 2*0.5, 3*0.5 + 4*0.5] = [2.0, 4.0].

  2. Step 2: Eliminate wrong options

    Options A, B, and D result from incorrect calculations of the dot product.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dot product: sum of element-wise products. [OK]
Hint: Dot product: sum of element-wise products. [OK]
Common Mistakes:
3. In the context of recurrent neural networks, which mechanism is specifically designed to address the vanishing gradient problem?
easy
A. Long Short-Term Memory (LSTM)
B. Batch Normalization
C. Weight Decay
D. Dropout

Solution

  1. Step 1: Identify the mechanism for vanishing gradient.

    LSTMs are specifically designed to handle vanishing gradients by using gating mechanisms.

  2. Step 2: Eliminate wrong options.

    Dropout (A) is for regularization, Batch Normalization (B) stabilizes learning, and Weight Decay (D) is for preventing overfitting.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    LSTM handles vanishing gradients with gates. [check]
Hint: LSTM handles vanishing gradients with gates. [check]
Common Mistakes:
4. For a task involving complex spatial hierarchies, which architecture is more suitable and why?
medium
A. A shallow network because it trains faster.
B. A shallow network because it requires less data.
C. A deep network because it can learn complex hierarchies.
D. A deep network because it always generalizes better.

Solution

  1. Step 1: Consider the task requirements

    Complex spatial hierarchies require the ability to learn multiple levels of abstraction, which deep networks are better suited for.

  2. Step 2: Eliminate wrong options

    A shallow network because it trains faster. is incorrect because training speed is not the primary concern. A shallow network because it requires less data. is incorrect because data requirements depend on the task complexity. A deep network because it always generalizes better. is incorrect because deep networks do not always generalize better.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Deep networks for complex hierarchies. [OK]
Hint: Deep networks for complex hierarchies. [OK]
Common Mistakes:
5. You are tasked with selecting features for a real-time prediction system with strict latency requirements. Which feature selection method is most suitable?
hard
A. Filter method
B. Embedded method
C. Wrapper method
D. Recursive feature elimination

Solution

  1. Step 1: Consider latency constraints.

    Embedded methods integrate feature selection into model training, optimizing for both accuracy and speed.

  2. Step 2: Eliminate wrong options.

    Filter methods do not consider model performance. Wrapper methods and recursive feature elimination are too slow for real-time systems.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Embedded = fast, integrated. [OK]
Hint: Embedded methods optimize speed and accuracy. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard