Bird
Raised Fist0

Random Forests - Bagging, Feature Randomness, OOD Robustness

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started immediately after a new feature was added last Monday.

A customer churn prediction model using Random Forests, deployed for a telecom company with 1 million active users.

Symptom
Model accuracy remains stable, but recall has dropped to 0 since the last feature update.
📡
Signals
metric
Recall on the test set
Recall dropped from 0.75 to 0.00 after the feature update
feature_importance
Feature importance ranking from the Random Forest model
New feature 'recent_plan_change' has the highest importance at 0.35
distribution
Distribution of the new feature 'recent_plan_change'
Highly skewed with 95% of values being 0
log
Model training logs
No errors, but warnings about feature imbalance
metric
Validation AUC
Validation AUC remains stable at 0.89
🔬
Hypotheses
The new feature 'recent_plan_change' is causing the model to overfit.
Supporting Signals
Feature importance rankingDistribution of the new feature
Verify: Check if the model's decision paths are overly reliant on 'recent_plan_change'.
from sklearn.tree import export_text; print(export_text(model.estimators_[0], feature_names=features))
The new feature is causing data leakage.
Supporting Signals
Feature importance rankingRecall drop
Verify: Check if the feature is derived from the target variable.
print(df[['recent_plan_change', 'churn']].corr())
🎯
Root Cause
The new feature 'recent_plan_change' is highly correlated with the target, causing overfitting.
Why It Happened

The feature was added without proper validation checks for correlation with the target.

🔧
Fix Strategy
ImmediateRemove the 'recent_plan_change' feature from the model.
Short-termRetrain the model without the problematic feature and validate performance.
Long-termImplement a feature validation pipeline to check for high correlation with the target before deployment.
🐍
Python Fix
Python
features = [f for f in features if f != 'recent_plan_change']; model.fit(X_train[features], y_train)
✗
Common Wrong Diagnoses
▼ Show
The model is underfitting due to insufficient data.
Why candidates think this: Recall drop might suggest the model isn't capturing patterns.
Why wrong: The stable accuracy and high feature importance indicate overfitting, not underfitting.
The model's hyperparameters need tuning.
Why candidates think this: Hyperparameter issues can cause performance drops.
Why wrong: The sudden drop in recall aligns with the feature update, not gradual hyperparameter drift.
📊
Monitoring to Add
  • Monitor feature importance changes post-deployment.
  • Set alerts for drastic drops in recall.
  • Track distribution changes of new features.
📋
Process Change

Introduce a mandatory feature validation step in the deployment pipeline.

🏗️
Architectural Change

Develop a feature engineering framework that automatically checks for target leakage.

💡
Key Insight
Feature importance spikes can indicate overfitting, especially when new features correlate with the target.

Practice

(1/5)
1. Given the following PyTorch code snippet, what will be the output shape of the gradient for W after backpropagation?
import torch
W = torch.tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
x = torch.tensor([1.0, 2.0])
y = W @ x
loss = y.sum()
loss.backward()
easy
A. (2, 2)
B. (2,)
C. (2, 1)
D. (1, 2)

Solution

  1. Step 1: Determine the gradient shape

    The gradient of W will have the same shape as W, which is (2, 2).

  2. Step 2: Eliminate wrong options

    (2,) is incorrect because it suggests a vector shape. (2, 1) and D are incorrect because they suggest mismatched dimensions.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Gradient shape matches tensor shape. [OK]
Hint: Gradient shape matches tensor shape. [OK]
Common Mistakes:
2. In a deep learning model, when is it more appropriate to use Layer Normalization instead of Batch Normalization?
easy
A. When the model is trained on a very large batch size.
B. When the model is trained on a very small batch size or online learning.
C. When the model has a very deep architecture.
D. When the model is trained on a balanced dataset.

Solution

  1. Step 1: Understand the context of normalization techniques.

    Batch Normalization relies on batch statistics, which can be unstable with small batch sizes. Layer Normalization uses layer statistics, making it more stable for small batches or online learning.

  2. Step 2: Eliminate wrong options.

    When the model is trained on a very large batch size. is incorrect because large batch sizes are suitable for Batch Normalization. When the model has a very deep architecture. is incorrect because depth doesn't directly affect the choice between these normalizations. When the model is trained on a balanced dataset. is irrelevant to normalization choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Layer Norm for small batches or online learning. [OK]
Hint: Layer Norm for small batches or online learning. [OK]
Common Mistakes:
3. 
import torch
import torch.nn as nn

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=0)
input_tensor = torch.ones((1, 1, 5, 5))
output_tensor = conv(input_tensor)
print(output_tensor.shape)
What is the output shape of the output_tensor?
easy
A. (1, 1, 5, 5)
B. (1, 1, 4, 4)
C. (1, 1, 2, 2)
D. (1, 1, 3, 3)

Solution

  1. Step 1: Calculate output dimensions

    Using the formula: ((W - F + 2P) / S) + 1, where W=5, F=3, P=0, S=1, the output size is ((5-3+0)/1)+1 = 3.

  2. Step 2: Eliminate wrong options

    (1, 1, 5, 5) is incorrect because the size doesn't remain the same. (1, 1, 4, 4) is incorrect because it assumes incorrect padding. (1, 1, 2, 2) is incorrect due to incorrect calculation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use output size formula for CNNs. [OK]
Hint: Use output size formula for CNNs. [OK]
Common Mistakes:
4. What will be the output shape of the following PyTorch code snippet? 
import torch
rnn = torch.nn.RNN(input_size=10, hidden_size=20, num_layers=2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
output, hn = rnn(input, h0)
print(output.shape)
easy
A. (5, 3, 10)
B. (3, 5, 20)
C. (2, 3, 20)
D. (5, 3, 20)

Solution

  1. Step 1: Understand RNN output shape.

    The output shape of an RNN is (seq_len, batch, hidden_size).

  2. Step 2: Apply to the given code.

    Here, seq_len=5, batch=3, hidden_size=20, so the output shape is (5, 3, 20).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    RNN output: (seq_len, batch, hidden_size). [check]
Hint: RNN output: (seq_len, batch, hidden_size). [check]
Common Mistakes:
5. When monitoring an ML model in production, which approach is better for detecting data drift and why?
medium
A. Monitor statistical properties of input features.
B. Use a static threshold on model accuracy.
C. Rely on user feedback to identify issues.
D. Increase the frequency of model retraining.

Solution

  1. Step 1: Identify effective drift detection

    Monitoring statistical properties helps detect subtle changes in input data.

  2. Step 2: Eliminate wrong options

    Use a static threshold on model accuracy. is reactive, not proactive. Rely on user feedback to identify issues. is unreliable and subjective. Increase the frequency of model retraining. addresses performance decay, not drift detection.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Monitor input stats for drift detection [OK]
Hint: Monitor input stats for drift detection [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard