Bird
Raised Fist0

Implement Decision Tree - Information Gain, Recursive Split

CODINGintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Information Gain vs Gini Impurity

When building a decision tree for a classification task, would you use Information Gain or Gini Impurity for splitting? Justify your choice.

📊
Comparison
DimensionInformation GainGini Impurity
accuracy on structured datahighmedium
training speedlowhigh
inference latencymediumhigh
interpretabilityhighmedium
robustness to noisemediumhigh
✅
Verdict
Default to Gini Impurity for larger, noisier datasets due to its computational efficiency and robustness. Switch to Information Gain for smaller, cleaner datasets where interpretability is key.
💡
Key Insight
Choose Information Gain for interpretability and insight on small, clean datasets; prefer Gini Impurity for efficiency and robustness on larger, noisier datasets.
⚠️
Interview Trap
❌ Wrong: Always use Information Gain because it provides the most informative splits.
This ignores the computational cost and potential bias towards features with more levels.
✅ Consider the dataset size, feature count, and noise level to decide between Information Gain and Gini Impurity.
Information Gain
"Maximizes the reduction in entropy, leading to more informative splits."
Weaknesses
  • Computationally expensive due to logarithmic calculations.
  • Can be biased towards features with more levels.
  • Less robust to noisy data compared to Gini Impurity.
Ideal When
  • n < 10K
  • features < 50
  • when interpretability is crucial
  • when the dataset is clean with minimal noise
ID3 algorithm uses Information Gain for building decision trees, often used in educational tools for teaching decision tree concepts.
Gini Impurity
"Computationally efficient and less sensitive to noise."
Weaknesses
  • May not always lead to the most informative splits.
  • Can be less interpretable due to less focus on entropy.
  • Biased towards features with fewer levels.
Ideal When
  • n > 10K
  • features > 50
  • when computational efficiency is crucial
  • when the dataset is noisy
CART algorithm uses Gini Impurity for decision trees, widely used in production systems for its efficiency.
🐍
Information Gain - Example
▼ Show
Python
import numpy as np

def information_gain(y, y_left, y_right):
    def entropy(y):
        hist = np.bincount(y)
        ps = hist / len(y)
        return -np.sum([p * np.log2(p) for p in ps if p > 0])
    parent_entropy = entropy(y)
    n = len(y)
    n_left, n_right = len(y_left), len(y_right)
    weighted_entropy = (n_left / n) * entropy(y_left) + (n_right / n) * entropy(y_right)
    return parent_entropy - weighted_entropy
🐍
Gini Impurity - Example
▼ Show
Python
import numpy as np

def gini_impurity(y, y_left, y_right):
    def gini(y):
        hist = np.bincount(y)
        ps = hist / len(y)
        return 1 - np.sum(ps ** 2)
    n = len(y)
    n_left, n_right = len(y_left), len(y_right)
    weighted_gini = (n_left / n) * gini(y_left) + (n_right / n) * gini(y_right)
    return weighted_gini
✅
Use Information Gain When
  • •n < 10K
  • •features < 50
  • •when interpretability is crucial
  • •when the dataset is clean with minimal noise
✅
Use Gini Impurity When
  • •n > 10K
  • •features > 50
  • •when computational efficiency is crucial
  • •when the dataset is noisy
↳
Nuances
  • For small datasets, Information Gain can provide more insightful splits despite its computational cost.
  • In noisy environments, Gini Impurity often provides more stable models.
  • When feature levels are unevenly distributed, both methods can exhibit bias, but Information Gain is more prone.

Practice

(1/5)
1. 
import numpy as np

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

vector1 = np.array([1, 2, 3])
vector2 = np.array([4, 5, 6])

result = cosine_similarity(vector1, vector2)
print(result)
easy
A. 0.9746318461970762
B. 0.9999999999999998
C. 1.0
D. 0.0

Solution

  1. Step 1: Calculate dot product and norms

    Dot product = 32, norm_a = 3.7416573867739413, norm_b = 8.774964387392123.

  2. Step 2: Compute cosine similarity

    Cosine similarity = 32 / (3.7416573867739413 * 8.774964387392123) = 0.9746318461970762.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dot product over product of norms. [OK]
Hint: Dot product over product of norms. [OK]
Common Mistakes:
2. 
import numpy as np

def dcg_at_k(r, k):
    r = np.asfarray(r)[:k]
    if r.size:
        return np.sum(r / np.log2(np.arange(2, r.size + 2)))
    return 0.0

relevance_scores = [3, 2, 3, 0, 1, 2]
k = 3
print(dcg_at_k(relevance_scores, k))
What does this code output?
easy
A. 5.2618595071429155
B. 4.2618595071429155
C. 3.2618595071429155
D. 6.2618595071429155

Solution

  1. Step 1: Understand DCG Calculation

    The DCG is calculated as the sum of relevance scores divided by log base 2 of their positions.

  2. Step 2: Calculate for k=3

    For k=3, the scores are [3, 2, 3]. The calculation is 3/1 + 2/log2(3) + 3/log2(4).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DCG uses log2 positions. [OK]
Hint: DCG uses log2 positions. [OK]
Common Mistakes:
3. What is the time complexity of computing the softmax function for an input array of size n?
easy
A. O(n^2)
B. O(n log n)
C. O(1)
D. O(n)

Solution

  1. Step 1: Analyze operations

    Softmax involves computing exponentials and a sum, both linear in n.

  2. Step 2: Determine complexity

    Each operation is O(n), so overall complexity is O(n).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Softmax is linear in input size [OK]
Hint: Softmax is linear in input size [OK]
Common Mistakes:
4. You implemented a normalization function for image data, but the model's performance dropped. What is the most likely cause?
medium
A. Normalization was applied after data augmentation.
B. Normalization was applied only to the training set.
C. Normalization was applied using incorrect mean and std values.
D. Normalization was applied using the global mean and std.

Solution

  1. Step 1: Identify normalization issues

    Incorrect mean and std values can skew the data distribution.

  2. Step 2: Eliminate other options

    Normalization was applied after data augmentation. is valid; Normalization was applied only to the training set. would cause a mismatch; Normalization was applied using the global mean and std. is standard practice.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check mean/std values for normalization. [OK]
Hint: Check mean/std values for normalization. [OK]
Common Mistakes:
5. A candidate implemented the cross-entropy loss function but it returns negative values. What is the most likely cause?
medium
A. The input probabilities are not normalized.
B. The log function is applied incorrectly.
C. The softmax function is not applied before the loss calculation.
D. The loss function is missing a regularization term.

Solution

  1. Step 1: Identify negative values cause

    Cross-entropy loss should be non-negative; negative values suggest log issues.

  2. Step 2: Check log application

    Log applied to values outside (0,1) can cause negatives.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Log of non-probabilities causes negatives [OK]
Hint: Log of non-probabilities causes negatives [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard