Bird
Raised Fist0

Implement Random Forest - Bagging, Feature Sampling

CODINGintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/4
Data Sampling

Data is split into multiple samples. Sample 1 contains 70% of the original data.

🎯 Understanding how data sampling works in bagging is crucial for Random Forest.
from sklearn.utils import resample
sample1 = resample(X, n_samples=int(0.7*len(X)), random_state=1)
Feature Sampling

For Tree 1, only a subset of features is selected: [f1, f3, f5].

🎯 Feature sampling helps in reducing correlation between trees.
features = [0, 2, 4]
X_sample1 = sample1[:, features]
Tree Training

Tree 1 is trained on Sample 1 with selected features. Accuracy is 85%.

🎯 Training individual trees on different samples and features is key to Random Forest.
from sklearn.tree import DecisionTreeClassifier
tree1 = DecisionTreeClassifier()
tree1.fit(X_sample1, y_sample1)
Ensemble Prediction

Predictions from all trees are combined. Final prediction is majority vote.

🎯 Combining predictions from multiple trees increases robustness.
predictions = [tree.predict(X_test) for tree in forest]
final_prediction = max(set(predictions), key=predictions.count)
📊
Diagram
Diagram
┌───────────────────────────────┐
│ Random Forest Construction    │
├───────────────────────────────┤
│ 1. Data Sample 1              │
│    ┌───────────────┐          │
│    │ Tree 1        │          │
│    └───────────────┘          │
│ 2. Data Sample 2              │
│    ┌───────────────┐          │
│    │ Tree 2        │          │
│    └───────────────┘          │
│ 3. Data Sample N              │
│    ┌───────────────┐          │
│    │ Tree N        │          │
│    └───────────────┘          │
└───────────────────────────────┘
Labels
  • Random Forest Construction: Overall process of building the forest
  • Data Sample 1: First subset of data used for training Tree 1
  • Tree 1: First decision tree in the forest
  • Data Sample N: N-th subset of data used for training Tree N
  • Tree N: N-th decision tree in the forest
🔑
Key Moments
▲
Q: Why is feature sampling important in Random Forest?
A: Feature sampling reduces correlation between trees, improving model robustness.
Q: How does bagging contribute to the performance of Random Forest?
A: Bagging reduces variance by training each tree on a different subset of data.
Q: What is the role of ensemble prediction in Random Forest?
A: Ensemble prediction combines outputs from all trees to make a final decision, enhancing accuracy.
Trace Summary

This diagram illustrates the construction of a Random Forest, highlighting the process of bagging and feature sampling. Each decision tree is trained on a different data sample and a subset of features, which helps in reducing correlation and variance. The ensemble prediction combines outputs from all trees to make a final decision, enhancing the model's accuracy and robustness. Understanding these steps is crucial for implementing Random Forest effectively.

Primary Focus

✓ Random Forest implementation with bagging and feature sampling

Practice

(1/5)
1. What is the time complexity of applying a dropout layer to an input of size n during training?
easy
A. O(1)
B. O(log n)
C. O(n^2)
D. O(n)

Solution

  1. Step 1: Analyze Dropout Operation

    Applying dropout involves generating a mask and element-wise multiplication, both O(n) operations.

  2. Step 2: Eliminate Wrong Options

    O(1) suggests constant time, which is incorrect. O(n^2) suggests quadratic time, which is excessive. O(log n) suggests logarithmic time, which is not applicable.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dropout involves element-wise operations. [OK]
Hint: Dropout involves element-wise operations. [OK]
Common Mistakes:
2. What does the following Python code snippet output when implementing Mini-Batch SGD with Momentum? 
import numpy as np

velocity = np.array([0.0, 0.0])
gradient = np.array([0.1, 0.2])
learning_rate = 0.01
momentum = 0.9

velocity = momentum * velocity - learning_rate * gradient
print(velocity)
easy
A. array([0.01, 0.02])
B. array([0.001, 0.002])
C. array([-0.001, -0.002])
D. array([-0.01, -0.02])

Solution

  1. Step 1: Calculate the velocity update

    The formula for velocity update is velocity = momentum * velocity - learning_rate * gradient. Given the initial velocity is zero, the update becomes velocity = -learning_rate * gradient.

  2. Step 2: Substitute the values

    Substitute learning_rate = 0.01 and gradient = [0.1, 0.2] into the formula: velocity = -0.01 * [0.1, 0.2] = [-0.001, -0.002].

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Velocity update is negative of learning rate times gradient [OK]
Hint: Velocity update is negative of learning rate times gradient [OK]
Common Mistakes:
3. You implemented Robust Scaling but your model's performance dropped. What is the most likely cause?
medium
A. Robust Scaling is not suitable for normally distributed data.
B. Robust Scaling requires data to be centered around zero.
C. Robust Scaling scales data to a range of [0, 1], which is incorrect.
D. Robust Scaling removes outliers, which are important for your model.

Solution

  1. Step 1: Understand Robust Scaling

    Robust Scaling uses the median and IQR, which reduces the influence of outliers.

  2. Step 2: Identify the Issue

    If outliers are important, reducing their influence can harm model performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Robust Scaling reduces outlier impact. [OK]
Hint: Robust Scaling reduces outlier impact. [OK]
Common Mistakes:
4. In the following implementation of scaled dot-product attention, the output is incorrect. What is the likely cause? 
import torch
import torch.nn.functional as F

queries = torch.randn(2, 3)
keys = torch.randn(2, 3)
values = torch.randn(2, 3)

scores = torch.matmul(queries, keys.T) / torch.sqrt(torch.tensor(3.0))
attention_weights = F.softmax(scores, dim=0)
output = torch.matmul(attention_weights, values)
medium
A. Incorrect scaling factor used.
B. Softmax applied on the wrong dimension.
C. Values tensor has incorrect shape.
D. Scores tensor is not normalized.

Solution

  1. Step 1: Identify the issue

    Softmax should be applied on the last dimension, not the first.

  2. Step 2: Eliminate other causes

    Scaling factor and tensor shapes are correct; normalization is handled by softmax.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Softmax on last dim for attention [OK]
Hint: Softmax on last dim for attention [OK]
Common Mistakes:
5. How can the calculation of precision, recall, and F1-score be optimized for a large confusion matrix?
hard
A. Use a more efficient data structure than a matrix.
B. Pre-compute and store results for common confusion matrices.
C. Use parallel processing to compute metrics for each class.
D. Approximate metrics using a subset of the data.

Solution

  1. Step 1: Identify bottlenecks

    Calculating metrics for each class can be parallelized.

  2. Step 2: Choose the best optimization strategy

    Parallel processing allows simultaneous computation, reducing time complexity.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Parallelize class-wise computations. [OK]
Hint: Parallelize class-wise computations. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard