Bird
Raised Fist0

Stratification Gone Wrong

CODINGbeginnerInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
Stratification Gone Wrong
The issue started immediately after the last deployment, which included a new data preprocessing script.

A customer churn prediction model for a telecom company, predicting churn for 1M customers monthly.

Symptom
Model accuracy dropped from 85% to 70% after the latest deployment.
📡
Signals
metric
Model accuracy on test set
Dropped from 85% to 70% after deployment
distribution
Class distribution in training vs test set
Training set: 70% non-churn, 30% churn; Test set: 90% non-churn, 10% churn
log
Deployment logs showing changes in data preprocessing
New stratification logic applied during train-test split
code
Code review of the train-test split function
Stratification logic incorrectly implemented
🔬
Hypotheses
The stratification logic in the train-test split is incorrect.
Supporting Signals
distributioncode
Verify: Check the stratification logic in the train-test split function.
import numpy as np
from collections import Counter

# Assuming y_train and y_test are the labels
train_distribution = Counter(y_train)
test_distribution = Counter(y_test)
print('Train distribution:', train_distribution)
print('Test distribution:', test_distribution)
The data preprocessing script introduced a bug affecting class balance.
Supporting Signals
log
Verify: Review the changes in the data preprocessing script.
with open('preprocessing_script.py', 'r') as file:
    lines = file.readlines()
    for line in lines:
        if 'stratification' in line:
            print(line)
🎯
Root Cause
The stratification logic was incorrectly implemented, leading to imbalanced class distribution in the test set.
Why It Happened

A recent change in the data preprocessing script introduced a bug in the stratification logic during the train-test split.

🔧
Fix Strategy
ImmediateRollback to the previous version of the data preprocessing script.
Short-termCorrect the stratification logic in the train-test split function.
Long-termImplement automated tests to verify class distribution after train-test split.
🐍
Python Fix
Python
import numpy as np

def stratified_train_test_split(X, y, test_size=0.2, random_state=None):
    np.random.seed(random_state)
    unique_classes, y_indices = np.unique(y, return_inverse=True)
    train_indices = []
    test_indices = []
    for class_index in range(len(unique_classes)):
        class_indices = np.where(y_indices == class_index)[0]
        np.random.shuffle(class_indices)
        n_test = int(len(class_indices) * test_size)
        test_indices.extend(class_indices[:n_test])
        train_indices.extend(class_indices[n_test:])
    return X[train_indices], X[test_indices], y[train_indices], y[test_indices]
✗
Common Wrong Diagnoses
▼ Show
The model architecture is flawed.
Why candidates think this: The drop in accuracy suggests a model issue.
Why wrong: The model was performing well before the deployment; the issue is with data distribution.
The feature engineering process introduced a bug.
Why candidates think this: Changes in preprocessing could affect model input.
Why wrong: The specific issue is with stratification, not general feature engineering.
📊
Monitoring to Add
  • Monitor class distribution in train and test sets after each split.
  • Add alerts for significant changes in model accuracy post-deployment.
  • Implement checks for data preprocessing script changes affecting class balance.
📋
Process Change

Require code reviews for all changes in data preprocessing scripts.

🏗️
Architectural Change

Use a data pipeline framework that includes built-in stratification checks.

💡
Key Insight
Stratification errors in train-test splits can drastically affect model performance by skewing class distributions.

Practice

(1/5)
1. What is the time complexity of computing a confusion matrix for a binary classification problem with n predictions?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Analyze the operation

    Computing a confusion matrix involves iterating over predictions once.

  2. Step 2: Determine complexity

    Each prediction is checked once, leading to O(n) complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Confusion matrix computation is linear in predictions. [OK]
Hint: Confusion matrix computation is linear in predictions. [OK]
Common Mistakes:
2. What does the following Python code snippet output when calculating the Intersection over Union (IoU) for two bounding boxes? 
import numpy as np

def iou(box1, box2):
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])
    intersection = max(0, x2 - x1) * max(0, y2 - y1)
    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
    union = area1 + area2 - intersection
    return intersection / union

box1 = [0, 0, 2, 2]
box2 = [1, 1, 3, 3]
print(iou(box1, box2))
easy
A. 0.14285714285714285
B. 0.3333333333333333
C. 0.5
D. 0.25

Solution

  1. Step 1: Calculate Intersection and Union

    The intersection area is 1 (from (1,1) to (2,2)). The union area is 7 (4 from box1 + 4 from box2 - 1 intersection).

  2. Step 2: Calculate IoU

    IoU = 1/7 = 0.14285714285714285.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Intersection over Union formula [OK]
Hint: Intersection over Union formula [OK]
Common Mistakes:
3. 
import numpy as np

confusion_matrix = np.array([[50, 10], [5, 35]])

precision = confusion_matrix[0, 0] / (confusion_matrix[0, 0] + confusion_matrix[1, 0])
recall = confusion_matrix[0, 0] / (confusion_matrix[0, 0] + confusion_matrix[0, 1])

print(f"Precision: {precision}, Recall: {recall}")
easy
A. Precision: 0.8333, Recall: 0.8333
B. Precision: 0.9091, Recall: 0.9091
C. Precision: 0.8333, Recall: 0.9091
D. Precision: 0.9091, Recall: 0.8333

Solution

  1. Step 1: Calculate Precision

    Precision is calculated as TP / (TP + FP) = 50 / (50 + 5) = 0.9091.

  2. Step 2: Calculate Recall

    Recall is calculated as TP / (TP + FN) = 50 / (50 + 10) = 0.8333.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Precision uses FP, Recall uses FN. [OK]
Hint: Precision uses FP, Recall uses FN. [OK]
Common Mistakes:
4. A Gaussian Naive Bayes model is giving poor performance on a dataset. Which of the following is the most likely cause?
medium
A. The model has too many parameters.
B. The features are not normally distributed.
C. The dataset is too small.
D. The features are highly correlated.

Solution

  1. Step 1: Identify Gaussian Naive Bayes assumptions

    Gaussian Naive Bayes assumes that features are normally distributed.

  2. Step 2: Evaluate each option

    The features are not normally distributed. is correct because non-normal distribution violates the model's assumptions. The model has too many parameters. is incorrect as GaussianNB is simple. The dataset is too small. is less likely as small datasets affect all models. The features are highly correlated. is incorrect as GaussianNB can handle correlated features.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GaussianNB assumes normal distribution. [OK]
Hint: GaussianNB assumes normal distribution. [OK]
Common Mistakes:
5. How can you optimize the computation of the ROC curve for a large dataset?
hard
A. Use a more efficient sorting algorithm for the scores.
B. Use a smaller batch size for computation.
C. Use a subset of data to approximate the ROC curve.
D. Use parallel processing to compute ROC for different thresholds.

Solution

  1. Step 1: Identify optimization strategies

    Parallel processing can speed up computation by handling different thresholds simultaneously.

  2. Step 2: Eliminate wrong options

    Use a smaller batch size for computation. doesn't address the core computation, Use a subset of data to approximate the ROC curve. may reduce accuracy, Use a more efficient sorting algorithm for the scores. is limited by sorting algorithm constraints.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Parallel processing speeds up ROC computation [OK]
Hint: Parallel processing speeds up ROC computation [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard