Bird
Raised Fist0

Cross-Validation Conundrum

CODINGintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Stratified Split Snafu
Started after the latest model deployment last Monday.

A sentiment analysis model predicting user satisfaction from reviews, deployed across multiple regions with varying class distributions.

Symptom
Model accuracy is 0.92 offline but drops to 0.75 in production.
📡
Signals
metric
Offline validation accuracy remains high at 0.92.
Offline validation accuracy is stable at 0.92.
metric
Production accuracy dropped significantly to 0.75.
Production accuracy is consistently at 0.75.
distribution
Class distribution in production data is skewed compared to training data.
Production data has 70% positive, 30% negative; training data was 50% positive, 50% negative.
log
Logs show stratified K-Fold was used during cross-validation.
Stratified K-Fold logs indicate balanced splits during validation.
code
Code review shows stratified K-Fold implementation.
Stratified K-Fold implemented with incorrect class balancing logic.
🔬
Hypotheses
The stratified K-Fold implementation is not correctly balancing classes.
Supporting Signals
Offline validation accuracy remains high at 0.92.Logs show stratified K-Fold was used during cross-validation.
Verify: Check the stratified K-Fold implementation for correct class balancing logic.
from collections import Counter

# Check class distribution in each fold
for train_idx, val_idx in StratifiedKFold(n_splits=5).split(X, y):
    train_classes = Counter(y[train_idx])
    val_classes = Counter(y[val_idx])
    print(f'Train: {train_classes}, Val: {val_classes}')
The production data distribution is significantly different from the training data.
Supporting Signals
Class distribution in production data is skewed compared to training data.
Verify: Compare class distributions between training and production datasets.
import matplotlib.pyplot as plt

# Plot class distributions
plt.hist([y_train, y_prod], label=['Train', 'Prod'], bins=2, alpha=0.7)
plt.legend()
plt.show()
🎯
Root Cause
The stratified K-Fold implementation incorrectly assumed equal class distribution, leading to unrepresentative validation splits.
Why It Happened

The implementation did not account for the natural class imbalance in the dataset, leading to overly optimistic validation results.

🔧
Fix Strategy
ImmediateRollback to the previous model version that used a simpler validation strategy.
Short-termCorrect the stratified K-Fold implementation to handle class imbalance properly.
Long-termImplement a robust validation framework that dynamically adjusts for class distribution changes.
🐍
Python Fix
Python
from sklearn.model_selection import StratifiedKFold

# Corrected stratified K-Fold implementation
kf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in kf.split(X, y):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    # Ensure class distribution is maintained
    assert len(set(y_train)) == len(set(y_test))
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The high offline accuracy suggests overfitting.
Why wrong: The validation process itself was flawed, not the model's generalization capability.
The feature set used in production is different from training.
Why candidates think this: A drop in production accuracy could indicate feature mismatch.
Why wrong: Feature sets were consistent; the issue was with validation splits not reflecting production data.
📊
Monitoring to Add
  • Monitor class distribution in validation splits to ensure they match production data.
  • Track validation accuracy against a holdout set that mirrors production distribution.
  • Alert on significant discrepancies between offline and online performance metrics.
  • Log and review class distribution changes in production data regularly.
📋
Process Change

Incorporate a review step for validation strategy changes in the deployment pipeline.

🏗️
Architectural Change

Design the system to automatically adjust validation strategies based on real-time data distribution analysis.

💡
Key Insight
Validation strategies must reflect production realities to ensure accurate performance estimation.

Practice

(1/5)
1. 
import numpy as np
from sklearn.metrics import confusion_matrix

y_true = [0, 1, 0, 1, 0, 1, 1]
y_pred = [0, 0, 0, 1, 1, 1, 1]

cm = confusion_matrix(y_true, y_pred)
print(cm)
What does the code output?
easy
A. [[2, 1], [1, 3]]
B. [[3, 1], [1, 2]]
C. [[2, 2], [1, 2]]
D. [[3, 0], [2, 2]]

Solution

  1. Step 1: Understand confusion matrix structure

    The confusion matrix is structured as [[TN, FP], [FN, TP]].

  2. Step 2: Calculate values

    TN=2 (true negatives), FP=1 (false positives), FN=1 (false negatives), TP=3 (true positives).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Confusion matrix is [[TN, FP], [FN, TP]]. [OK]
Hint: Confusion matrix is [[TN, FP], [FN, TP]]. [OK]
Common Mistakes:
2. 
import numpy as np

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

vector1 = np.array([1, 2, 3])
vector2 = np.array([4, 5, 6])

result = cosine_similarity(vector1, vector2)
print(result)
easy
A. 0.9746318461970762
B. 0.9999999999999998
C. 1.0
D. 0.0

Solution

  1. Step 1: Calculate dot product and norms

    Dot product = 32, norm_a = 3.7416573867739413, norm_b = 8.774964387392123.

  2. Step 2: Compute cosine similarity

    Cosine similarity = 32 / (3.7416573867739413 * 8.774964387392123) = 0.9746318461970762.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dot product over product of norms. [OK]
Hint: Dot product over product of norms. [OK]
Common Mistakes:
3. What does the following Python code snippet output when implementing a Gaussian Naive Bayes classifier?
import numpy as np
from sklearn.naive_bayes import GaussianNB
X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([0, 1, 0])
model = GaussianNB()
model.fit(X, y)
print(model.predict([[2, 3]]))
easy
A. [0]
B. [1]
C. [0, 1]
D. [1, 0]

Solution

  1. Step 1: Understand GaussianNB prediction

    The GaussianNB model predicts the class with the highest posterior probability.

  2. Step 2: Calculate probabilities

    For input [2, 3], the model calculates probabilities for each class and chooses the one with the highest probability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GaussianNB predicts based on highest posterior probability. [OK]
Hint: GaussianNB predicts based on highest posterior probability. [OK]
Common Mistakes:
4. 
import numpy as np

def euclidean_distance(a, b):
    return np.sqrt(np.sum((a - b) ** 2))

vector1 = np.array([1, 2, 3])
vector2 = np.array([4, 5])

result = euclidean_distance(vector1, vector2)
print(result)
medium
A. Returns NaN due to broadcasting
B. Throws a ValueError due to shape mismatch
C. Returns 0.0
D. Returns a large number due to incorrect subtraction

Solution

  1. Step 1: Check vector shapes

    vector1 has shape (3,), vector2 has shape (2,).

  2. Step 2: Identify operation failure

    Subtraction fails due to shape mismatch, raising a ValueError.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Shape mismatch leads to ValueError. [OK]
Hint: Shape mismatch leads to ValueError. [OK]
Common Mistakes:
5. In a dataset with highly imbalanced classes, how should you implement train-test split with stratification to ensure robust model evaluation?
medium
A. Ensure stratification is based on the minority class
B. Use a very small test size to capture rare classes
C. Use random splitting to avoid bias
D. Increase the test size to balance classes

Solution

  1. Step 1: Understand stratification in imbalanced datasets

    Stratification should focus on maintaining the distribution of the minority class.

  2. Step 2: Apply correct strategy

    Ensuring stratification based on the minority class helps in capturing its distribution in both sets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stratify on minority class for balance. [OK]
Hint: Stratify on minority class for balance. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard