Bird
Raised Fist0

The Feature Importance Anomaly

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Anomaly
Started immediately after the deployment of a new feature set last Friday.

A recommendation system model serving 5M users daily, predicting user engagement with content.

Symptom
Model accuracy remains stable, but recall has dropped to 0 since the last feature update.
📡
Signals
metric
Recall metric from the model evaluation dashboard
Recall dropped from 0.65 to 0.00 after the feature update
feature_importance
Feature importance ranking from the model's explainability tool
Feature 'user_last_active_days' importance dropped from rank #2 (0.18) to rank #15 (0.02) post-deploy
distribution
Distribution of the 'user_last_active_days' feature in the input data
Feature 'user_last_active_days' has a constant value of 0 for all inputs post-deploy
log
Deployment logs from the feature engineering pipeline
Logs show a new feature transformation script was deployed last Friday
metric
AUC metric from the model evaluation dashboard
AUC remains stable at 0.85, indicating overall model performance is unaffected
🔬
Hypotheses
The new feature transformation script introduced a bug affecting 'user_last_active_days'.
Supporting Signals
Feature 'user_last_active_days' importance droppedDistribution of 'user_last_active_days' is constant
Verify: Check the feature transformation script for errors affecting 'user_last_active_days'.
import pandas as pd

data = pd.read_csv('input_data.csv')
print(data['user_last_active_days'].unique())
The model is overfitting to the training data due to a lack of diversity in 'user_last_active_days'.
Supporting Signals
Feature 'user_last_active_days' importance droppedRecall dropped to 0
Verify: Analyze the training data distribution for 'user_last_active_days'.
import pandas as pd

data = pd.read_csv('training_data.csv')
print(data['user_last_active_days'].describe())
🎯
Root Cause
The feature transformation script incorrectly set 'user_last_active_days' to 0 for all inputs.
Why It Happened

A bug in the new feature transformation script deployed last Friday.

🔧
Fix Strategy
ImmediateRollback to the previous version of the feature transformation script.
Short-termCorrect the bug in the feature transformation script and redeploy.
Long-termImplement automated tests for feature transformation scripts to catch such errors before deployment.
🐍
Python Fix
Python
def transform_features(data):
    # Correct the transformation logic
    data['user_last_active_days'] = data['user_last_active_days'].apply(lambda x: max(x, 0))
    return data

# Apply the fix
corrected_data = transform_features(data)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to a lack of regularization.
Why candidates think this: The drop in recall suggests overfitting, which is a common issue.
Why wrong: The stable AUC and the specific feature importance drop indicate a feature-specific issue, not general overfitting.
The data pipeline introduced noise into the input data.
Why candidates think this: Noise in data can lead to performance drops.
Why wrong: The constant value in 'user_last_active_days' is not noise but a systematic error.
📊
Monitoring to Add
  • Monitor distribution of key features like 'user_last_active_days' post-deployment.
  • Set up alerts for drastic changes in feature importance rankings.
  • Implement checks for zero variance in feature distributions.
📋
Process Change

Introduce a code review process specifically for feature transformation scripts.

🏗️
Architectural Change

Adopt a feature store that includes validation checks for feature transformations.

💡
Key Insight
Feature transformation bugs can silently degrade model performance, making systematic monitoring essential.

Practice

(1/5)
1. 
from sklearn.tree import DecisionTreeClassifier
import numpy as np

X = np.array([[0, 0], [1, 1], [1, 0], [0, 1]])
y = np.array([0, 1, 1, 0])

clf = DecisionTreeClassifier(criterion='gini')
clf.fit(X, y)

print(clf.predict([[0, 0], [1, 1]]))
What is the output of the above code?
easy
A. [0, 0]
B. [1, 0]
C. [0, 1]
D. [1, 1]

Solution

  1. Step 1: Understand the training data.

    The data is perfectly separable, and the decision tree will learn to classify based on the given labels.

  2. Step 2: Predict using the trained model.

    For input [0, 0], the label is 0, and for [1, 1], the label is 1.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check training labels for predictions. [OK]
Hint: Check training labels for predictions. [OK]
Common Mistakes:
2. Given the following Python code snippet, what is the output?
import numpy as np
X = np.array([[1, 2], [3, 4]])
Y = X.T
print(Y.shape)
easy
A. (1, 4)
B. (4, 1)
C. (2, 2)
D. (2, 1)

Solution

  1. Step 1: Understand the operation.

    The code transposes the matrix X, which is a 2x2 matrix.

  2. Step 2: Determine the shape after transpose.

    The transpose of a 2x2 matrix is still 2x2.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Transpose keeps square matrix shape. [OK]
Hint: Transpose keeps square matrix shape. [OK]
Common Mistakes:
3. Given the following Python code using BPE tokenization, what is the output?
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace

# Initialize a BPE tokenizer
bpe_tokenizer = Tokenizer(BPE())
bpe_tokenizer.pre_tokenizer = Whitespace()

# Train the tokenizer
bpe_tokenizer.train(["hello world", "hello there"], vocab_size=10)

# Encode a sentence
output = bpe_tokenizer.encode("hello world")
print(output.tokens)
easy
A. ['h', 'e', 'l', 'l', 'o', 'world']
B. ['he', 'llo', 'world']
C. ['hello', 'wo', 'rld']
D. ['hello', 'world']

Solution

  1. Step 1: Understand BPE tokenization with whitespace pre-tokenization.

    BPE with whitespace pre-tokenization will initially split on spaces.

  2. Step 2: Analyze the output of the encode method.

    The sentence 'hello world' is tokenized as ['hello', 'world'] due to whitespace splitting.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    BPE with whitespace splits on spaces first. [OK]
Hint: BPE with whitespace splits on spaces first. [OK]
Common Mistakes:
4. You are choosing between a decision tree and a linear regression model for a dataset with a complex, non-linear relationship. Which model should you choose and why?
medium
A. Linear regression, because it is less likely to overfit.
B. Linear regression, because it is computationally cheaper.
C. Decision tree, because it can capture non-linear relationships.
D. Decision tree, because it has lower bias.

Solution

  1. Step 1: Analyze the dataset characteristics

    The dataset has a complex, non-linear relationship.

  2. Step 2: Choose the appropriate model

    Decision trees can capture non-linear relationships, making them suitable for this dataset.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Decision trees capture non-linear patterns. [OK]
Hint: Decision trees capture non-linear patterns. [OK]
Common Mistakes:
5. When designing a CNN for image classification, why might you choose to use a smaller receptive field in the initial layers?
medium
A. To capture more global features early on
B. To reduce the computational cost of the network
C. To capture fine-grained local features
D. To increase the depth of the network without increasing parameters

Solution

  1. Step 1: Understand receptive fields

    Smaller receptive fields in initial layers help capture fine-grained local features, which are crucial for understanding detailed patterns.

  2. Step 2: Eliminate wrong options

    To capture more global features early on is incorrect because global features are captured in deeper layers. To reduce the computational cost of the network is incorrect as smaller receptive fields don't necessarily reduce computational cost. To increase the depth of the network without increasing parameters is incorrect because depth and parameters are not directly related to receptive field size.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Smaller fields capture local details. [OK]
Hint: Smaller fields capture local details. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard