Bird
Raised Fist0

Decision Trees - Information Gain, Gini Impurity, Pruning

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Pruning Paradox
Started immediately after the latest model deployment last Monday.

A decision tree model used for predicting customer churn in a telecom company, processing 500k records monthly.

Symptom
Model accuracy is stable at 0.85, but recall has dropped to 0.0 after the last model update.
📡
Signals
metric
Recall on the test set
Recall dropped from 0.72 to 0.0 after the last deployment
feature_importance
Feature importance ranking
Feature 'contract_length' importance dropped from 0.25 to 0.01
distribution
Distribution of predicted classes
All predictions are 'not churn' after the last deployment
log
Model training logs
Pruning reduced tree depth from 15 to 3
code
Pruning configuration in the model code
Pruning threshold set to 0.05
🔬
Hypotheses
Pruning is too aggressive, leading to underfitting.
Supporting Signals
Recall dropped to 0.0Tree depth reduced from 15 to 3
Verify: Check if the pruning threshold is set too high, causing loss of important branches.
print(tree.get_depth())
Feature 'contract_length' is crucial and its importance drop affected recall.
Supporting Signals
Feature 'contract_length' importance droppedAll predictions are 'not churn'
Verify: Analyze the impact of 'contract_length' on predictions.
df['contract_length'].corr(df['churn'])
🎯
Root Cause
Pruning threshold was set too high, causing the model to underfit and lose critical decision paths.
Why It Happened

A recent change in the pruning configuration was not properly validated, leading to excessive pruning.

🔧
Fix Strategy
ImmediateRollback to the previous model version with less aggressive pruning.
Short-termAdjust the pruning threshold to a more conservative value and retrain the model.
Long-termImplement a validation step for pruning configuration changes to ensure they don't degrade model performance.
🐍
Python Fix
Python
tree = DecisionTreeClassifier(max_depth=None, min_impurity_decrease=0.01)
tree.fit(X_train, y_train)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to too many features.
Why candidates think this: Overfitting is a common issue with decision trees.
Why wrong: The model is actually underfitting due to excessive pruning, not overfitting.
Data leakage from training to test set.
Why candidates think this: Data leakage can cause unexpected drops in performance metrics.
Why wrong: The issue is with model configuration, not data leakage, as evidenced by pruning logs.
📊
Monitoring to Add
  • Monitor recall on a validation set after each deployment.
  • Track changes in feature importance rankings post-deployment.
  • Alert on significant drops in predicted positive class distribution.
📋
Process Change

Introduce a mandatory review process for changes in model hyperparameters, especially those affecting pruning.

🏗️
Architectural Change

Develop a robust A/B testing framework to evaluate model changes in a controlled environment before full deployment.

💡
Key Insight
Aggressive pruning can lead to underfitting, causing critical decision paths to be lost and recall to drop to zero.

Practice

(1/5)
1. Given the following Python code using Word2Vec, what is the output shape of the word vector for 'king'? 
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, workers=4)
vector = model.wv['king']
easy
A. (100, 1)
B. (1, 100)
C. (100,)
D. (1,)

Solution

  1. Step 1: Understand Word2Vec output

    The Word2Vec model returns a 1D array of the specified vector size for each word.

  2. Step 2: Determine the shape

    The vector size is set to 100, so the output shape is (100,).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Word2Vec outputs 1D vector of size [OK]
Hint: Word2Vec outputs 1D vector of size [OK]
Common Mistakes:
2. Given the following PyTorch code snippet for a simple GNN layer, what will be the output shape if the input node features have shape (5, 10) and the adjacency matrix is (5, 5)?
import torch.nn as nn
import torch

class SimpleGNNLayer(nn.Module):
    def __init__(self, in_features, out_features):
        super(SimpleGNNLayer, self).__init__()
        self.linear = nn.Linear(in_features, out_features)

    def forward(self, x, adj):
        return torch.matmul(adj, self.linear(x))

x = torch.randn(5, 10)
adj = torch.eye(5)
gnn_layer = SimpleGNNLayer(10, 20)
output = gnn_layer(x, adj)
easy
A. (5, 20)
B. (5, 10)
C. (10, 20)
D. (5, 5)

Solution

  1. Step 1: Understand the operations

    The linear layer transforms the input features from 10 to 20 dimensions.

  2. Step 2: Apply the adjacency matrix

    The adjacency matrix is used to aggregate features, maintaining the node count.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Linear layer changes feature dim, adj keeps node count. [check]
Hint: Linear layer changes feature dim, adj keeps node count. [check]
Common Mistakes:
3. A model is showing unexpectedly high validation accuracy but low test accuracy. What is the most likely cause?
medium
A. The model is underfitting the training data.
B. The test set is too similar to the training set.
C. The model has too many parameters.
D. The validation set is too small and not representative.

Solution

  1. Step 1: Diagnose the issue

    High validation accuracy but low test accuracy suggests the validation set may not be representative.

  2. Step 2: Eliminate wrong options

    The model is underfitting the training data. would affect both validation and test accuracy, The test set is too similar to the training set. would not cause low test accuracy, and The model has too many parameters. would affect both validation and test accuracy similarly.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Small validation set = misleading accuracy [OK]
Hint: Small validation set = misleading accuracy [OK]
Common Mistakes:
4. For a regression task with a large number of outliers, which metric would be more appropriate to use?
medium
A. Mean Squared Error (MSE)
B. Huber Loss
C. Mean Absolute Percentage Error (MAPE)
D. Root Mean Squared Error (RMSE)

Solution

  1. Step 1: Consider the impact of outliers on metrics.

    Huber Loss is less sensitive to outliers than MSE and RMSE.

  2. Step 2: Eliminate wrong options.

    MSE and RMSE amplify outliers due to squaring. MAPE is not suitable for zero or near-zero values.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Huber Loss balances sensitivity to outliers. [OK]
Hint: Huber Loss balances sensitivity to outliers. [OK]
Common Mistakes:
5. You are tasked with designing a GNN for a dynamic graph where nodes and edges change over time. Which approach would best handle this scenario?
hard
A. Adopt a dynamic GNN that updates with graph changes.
B. Implement a GAT with a fixed adjacency matrix.
C. Use a standard RNN to capture temporal changes.
D. Use a static GCN and retrain the model periodically.

Solution

  1. Step 1: Understand the graph dynamics

    Dynamic graphs require models that can adapt to changes in nodes and edges.

  2. Step 2: Evaluate each approach

    Static GCNs and fixed adjacency matrices cannot adapt to changes, and RNNs are not designed for graph structures.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dynamic GNNs adapt to graph changes. [check]
Hint: Dynamic GNNs adapt to graph changes. [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard