Bird
Raised Fist0

Decision Trees - Information Gain, Gini Impurity, Pruning

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Information Gain vs Gini Impurity

When building a decision tree for a customer churn prediction model, would you use Information Gain or Gini Impurity to split nodes? Justify your choice.

📊
Comparison
DimensionInformation GainGini Impurity
accuracy on small datasetshighmedium
computation speedlowhigh
interpretabilityhighmedium
robustness to noiselowhigh
scalabilitymediumhigh
✅
Verdict
Default to Gini Impurity for datasets larger than 10K samples or with high feature counts due to its computational efficiency. Use Information Gain for smaller, cleaner datasets where interpretability is a priority.
💡
Key Insight
The choice between Information Gain and Gini Impurity hinges on the trade-off between computational efficiency and the informativeness of splits.
⚠️
Interview Trap
❌ Wrong: Always use Information Gain because it provides more informative splits.
Ignoring the computational cost and potential overfitting on large datasets.
✅ Consider the trade-off between computational efficiency and the informativeness of splits, especially in relation to dataset size and feature count.
Information Gain
"Maximizes reduction in entropy, leading to more informative splits."
Weaknesses
  • Computationally expensive due to logarithmic calculations.
  • Prone to overfitting with small datasets.
  • Less effective with noisy data.
Ideal When
  • n < 10K
  • features < 50
  • low label noise
Amazon uses Information Gain in decision trees for product recommendation systems where interpretability is crucial.
Gini Impurity
"Faster computation due to simpler calculations, making it suitable for larger datasets."
Weaknesses
  • May lead to less informative splits compared to Information Gain.
  • Can be biased towards features with more levels.
  • Less interpretable in terms of information theory.
Ideal When
  • n > 10K
  • features > 50
  • high label noise
Facebook uses Gini Impurity in decision trees for large-scale ad targeting where speed is critical.
🐍
Information Gain - Example
▼ Show
Python
from math import log2

def information_gain(parent_entropy, left_entropy, right_entropy, left_size, right_size):
    total_size = left_size + right_size
    weighted_entropy = (left_size / total_size) * left_entropy + (right_size / total_size) * right_entropy
    return parent_entropy - weighted_entropy
🐍
Gini Impurity - Example
▼ Show
Python
def gini_impurity(groups, classes):
    n_instances = float(sum([len(group) for group in groups]))
    gini = 0.0
    for group in groups:
        size = float(len(group))
        if size == 0:
            continue
        score = 0.0
        for class_val in classes:
            p = [row[-1] for row in group].count(class_val) / size
            score += p * p
        gini += (1.0 - score) * (size / n_instances)
    return gini
✅
Use Information Gain When
  • •n < 10K
  • •features < 50
  • •low label noise
✅
Use Gini Impurity When
  • •n > 10K
  • •features > 50
  • •high label noise
↳
Nuances
  • Information Gain can overfit on small datasets due to its sensitivity to entropy changes.
  • Gini Impurity may favor features with more categories, which can skew results if not handled properly.
  • In high-dimensional spaces, the computational advantage of Gini Impurity becomes more pronounced.

Practice

(1/5)
1. In the context of Word2Vec, which of the following best describes the purpose of the 'skip-gram' model?
easy
A. Generates a sequence of words from a given context.
B. Predicts the next word in a sequence given the previous words.
C. Clusters words into predefined categories.
D. Predicts surrounding words given a target word.

Solution

  1. Step 1: Understand skip-gram model purpose

    The skip-gram model in Word2Vec is designed to predict surrounding words given a target word.

  2. Step 2: Eliminate wrong options

    Predicts the next word in a sequence given the previous words. describes a language model task, Clusters words into predefined categories. is about clustering, and Generates a sequence of words from a given context. is about sequence generation, none of which are the purpose of skip-gram.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Skip-gram predicts context from target [OK]
Hint: Skip-gram predicts context from target [OK]
Common Mistakes:
2. 
import numpy as np

y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])

mae = np.mean(np.abs(y_true - y_pred))
print(mae)
What is the output of this code snippet?
easy
A. 0.625
B. 0.5
C. 1.0
D. 1.25

Solution

  1. Step 1: Calculate the absolute errors.

    Errors are [0.5, 0.5, 0, 1].

  2. Step 2: Compute the mean of absolute errors.

    Mean is (0.5 + 0.5 + 0 + 1) / 4 = 0.625.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sum errors, divide by count. [OK]
Hint: Sum absolute errors, divide by count. [OK]
Common Mistakes:
3. A continually learning model is showing degraded performance over time. What is the most likely cause?
medium
A. The model is overfitting to recent data.
B. The model's architecture is too simple.
C. The model is not receiving enough new data.
D. The model's learning rate is too high.

Solution

  1. Step 1: Identify symptoms of degraded performance.

    Continual learning models can overfit to recent data, especially if not managed properly.

  2. Step 2: Eliminate other causes.

    The model is not receiving enough new data. is incorrect because lack of data would not cause degradation but stagnation. The model's architecture is too simple. is incorrect because architecture simplicity would limit initial performance, not degrade it over time. The model's learning rate is too high. is incorrect because a high learning rate would cause instability, not gradual degradation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Degradation often means overfitting. [OK]
Hint: Degradation often means overfitting. [OK]
Common Mistakes:
4. When deploying a model to production, which trade-off is most critical to consider?
medium
A. Model accuracy vs. training time
B. Model complexity vs. inference latency
C. Data size vs. model interpretability
D. Feature engineering vs. hyperparameter tuning

Solution

  1. Step 1: Identify the key trade-off in production.

    In production, inference latency is crucial as it affects user experience.

  2. Step 2: Eliminate less relevant trade-offs.

    Model accuracy vs. training time is more relevant during development, not deployment. Data size vs. interpretability and feature engineering vs. hyperparameter tuning are less critical in production.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Latency impacts user experience in production. [OK]
Hint: Latency impacts user experience in production. [OK]
Common Mistakes:
5. You need to design a model for a task involving both spatial and temporal hierarchies. Which approach is most suitable?
hard
A. Use a deep CNN-RNN hybrid to capture both spatial and temporal hierarchies.
B. Use a deep CNN for spatial features and ignore temporal features.
C. Use a shallow RNN for temporal features and ignore spatial features.
D. Use a shallow CNN for spatial features and a separate RNN for temporal features.

Solution

  1. Step 1: Identify task requirements

    The task requires capturing both spatial and temporal hierarchies, which suggests a hybrid approach.

  2. Step 2: Eliminate wrong options

    Use a shallow CNN for spatial features and a separate RNN for temporal features. is incorrect because separate models may not capture interactions. Use a shallow RNN for temporal features and ignore spatial features. is incorrect because it ignores spatial features. Use a deep CNN for spatial features and ignore temporal features. is incorrect because it ignores temporal features.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Hybrid models for spatial-temporal tasks. [OK]
Hint: Hybrid models for spatial-temporal tasks. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard