Feature importance explanation in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is feature importance in machine learning?

Feature importance tells us how much each input feature helps the model make good predictions. It shows which features matter most.

beginner

Name one simple way to calculate feature importance.

One simple way is to look at how much the model's accuracy drops when we shuffle or remove a feature. Bigger drops mean more important features.

beginner

Why is understanding feature importance useful?

It helps us trust the model, find key factors in data, and sometimes improve the model by focusing on important features.

intermediate

What is a common method to get feature importance from tree-based models?

Tree-based models like Random Forests use how much each feature reduces error or impurity in splits to measure importance.

beginner

What does it mean if a feature has zero importance?

It means the feature does not help the model make better predictions and can often be removed without hurting performance.

Which feature importance method measures importance by shuffling feature values and checking accuracy drop?

A Coefficient magnitude

B Permutation importance

C PCA

D Gradient descent

Permutation importance shuffles a feature's values and measures how much the model's accuracy decreases.

In tree models, feature importance is often based on:

A The feature's mean value

B The feature's data type

C The feature's correlation with the target

D How often a feature is used to split nodes and how much it reduces error

Tree models calculate importance by how much a feature helps split data to reduce error or impurity.

If a feature has zero importance, what should you consider?

A Adding more features like it

B Doubling its values

C Removing it from the model

D Ignoring the model

Features with zero importance do not help predictions and can be removed to simplify the model.

Why might feature importance help in real life?

A To understand which factors affect outcomes the most

B To make the model slower

C To hide data from users

D To increase data size

Feature importance helps identify key factors influencing predictions, aiding understanding and trust.

Which of these is NOT a way to get feature importance?

A Feature scaling

B Tree split importance

C Permutation importance

D Coefficient size in linear models

Feature scaling changes feature values but does not measure importance.

Explain in your own words what feature importance means and why it matters.

Think about which features help the model make better guesses.

Describe one method to calculate feature importance and how you would interpret its results.

Consider shuffling feature values and checking model accuracy.

Practice

1. What does feature importance tell us in a machine learning model?

easy

A. The time taken to train the model

B. Which features most affect the model's predictions

C. The accuracy of the model on test data

D. The total number of features in the dataset

Solution

1. Step 1: Understand the meaning of feature importance

   Feature importance measures how much each input feature influences the model's output predictions.

2. Step 2: Compare options with definition

   Only Which features most affect the model's predictions correctly states that feature importance shows which features most affect predictions.

3. Final Answer:

   Which features most affect the model's predictions -> Option B

4. Quick Check:

   Feature importance = Most influential features [OK]

Hint: Feature importance ranks features by prediction impact [OK]

Common Mistakes:

• Confusing feature count with importance
• Mixing accuracy with feature importance
• Thinking it measures training time

2. Which of the following is the correct way to get feature importance from a trained Random Forest model in Python using scikit-learn?

easy

A. model.feature_importances_

B. model.get_feature_importance()

C. model.feature_importance_

D. model.importance_features

Solution

1. Step 1: Recall scikit-learn Random Forest attribute

   The correct attribute to access feature importance is feature_importances_ with an 's' and underscore at the end.

2. Step 2: Check each option

   model.feature_importances_ matches the exact attribute name. Others are incorrect or do not exist.

3. Final Answer:

   model.feature_importances_ -> Option A

4. Quick Check:

   Random Forest uses feature_importances_ [OK]

Hint: Remember the plural 'importances' with trailing underscore [OK]

Common Mistakes:

• Using singular 'feature_importance_'
• Calling a non-existent method
• Mixing attribute names

3. You have a dataset with features: Age, Income, and Number of Purchases. After training a Random Forest, the feature importances are: Age=0.1, Income=0.7, Number of Purchases=0.2. What does this tell you about your model?

easy

A. Age is the most important feature influencing predictions

B. All features are equally important

C. Number of Purchases has no effect on predictions

D. Income is the most important feature influencing predictions

Solution

1. Step 1: Interpret feature importance values

   Higher values mean stronger influence on model predictions. Income has 0.7, which is highest.

2. Step 2: Compare other features

   Age (0.1) and Number of Purchases (0.2) have lower importance, so less influence.

3. Final Answer:

   Income is the most important feature influencing predictions -> Option D

4. Quick Check:

   Highest importance = most influential feature [OK]

Hint: Highest importance value shows key feature [OK]

Common Mistakes:

• Thinking low values mean no effect
• Assuming equal importance despite values
• Confusing feature names

4. Given this code snippet:

from sklearn.ensemble import RandomForestClassifier
X = [[1, 2], [3, 4], [5, 6], [7, 8]]
y = [0, 1, 0, 1]
model = RandomForestClassifier(random_state=0)
model.fit(X, y)
importances = model.feature_importances_
print(importances)

What is the expected output?

medium

A. [0.5 0.5]

B. [0.7 0.3]

C. [0.0 1.0]

D. An error because data is too small

Solution

1. Step 1: Understand the dataset and model

   The dataset has two features with balanced classes. RandomForestClassifier will split importance roughly evenly.

2. Step 2: Predict feature importance values

   With simple data and no strong bias, feature importance will be close to equal, around 0.5 each.

3. Final Answer:

   [0.5 0.5] -> Option A

4. Quick Check:

   Balanced features = equal importance [OK]

Hint: Small balanced data leads to roughly equal importances [OK]

Common Mistakes:

• Expecting exact 0 or 1 values
• Assuming error due to small data
• Thinking importance sums to more than 1

5. You run this code but get an AttributeError:

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.feature_importances_

What is the likely cause?

medium

A. You need to import a different library for feature importance

B. RandomForestClassifier does not have feature_importances_ attribute

C. The model is not trained yet before accessing feature_importances_

D. The attribute name is misspelled

Solution

1. Step 1: Check model usage

   The model is created but not trained with fit() before accessing feature_importances_.

2. Step 2: Understand attribute availability

   Feature importance is only available after training. Accessing it before fit causes AttributeError.

3. Final Answer:

   The model is not trained yet before accessing feature_importances_ -> Option C

4. Quick Check:

   Train model before feature_importances_ [OK]

Hint: Always call fit() before feature_importances_ [OK]

Common Mistakes:

• Assuming attribute doesn't exist
• Thinking import is missing
• Misspelling attribute name