Feature importance in regression in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Feature importance in regression

Which metric matters for this concept and WHY

For feature importance in regression, the key metric is R-squared (R²) and Mean Squared Error (MSE). R² tells us how well the features explain the variation in the target. MSE shows how close the predictions are to actual values. When checking feature importance, we look at how these metrics change when we remove or shuffle a feature. A big drop in R² or rise in MSE means that feature is important.

Confusion matrix or equivalent visualization (ASCII)

Regression does not use a confusion matrix because it predicts continuous values, not categories. Instead, we use error metrics like MSE or visualize feature importance with bar charts showing the impact of each feature on prediction error.

Feature Importance Example (higher value means more important):

Feature A: |||||||||||| (0.35)
Feature B: |||||||      (0.20)
Feature C: |||          (0.10)
Feature D: |            (0.05)

These numbers represent how much the error increases when the feature is removed or shuffled.
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

In regression, we don't have precision or recall. Instead, the tradeoff is between model complexity and feature selection. Using too many features can cause overfitting (model fits training data well but fails on new data). Using too few features can cause underfitting (model misses important patterns). Feature importance helps find the right balance by showing which features truly help prediction.

Example: If a house price model includes many features but only a few like size and location matter, removing less important features keeps the model simple and accurate.

What "good" vs "bad" metric values look like for this use case

Good feature importance results show:

• Clear ranking of features by importance.
• Removing top features causes a noticeable drop in R² or rise in MSE.
• Low importance features have little effect on metrics when removed.

Bad results show:

• All features have similar importance, making it hard to tell which matter.
• Removing features does not change R² or MSE, meaning features may be irrelevant or model is not sensitive.
• Feature importance values are unstable across different data samples.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Data leakage: If features leak information from the future or target, importance looks artificially high.
• Overfitting: Complex models may assign high importance to noise features, misleading interpretation.
• Correlation: Highly correlated features can share importance, making it hard to separate their effects.
• Ignoring metric changes: Looking only at importance scores without checking how metrics like R² or MSE change can mislead.

Self-check question

Your regression model has an R² of 0.85. When you remove Feature X, R² drops to 0.50. Is Feature X important? Why?

Answer: Yes, Feature X is important because removing it causes a large drop in R², meaning the model loses much of its ability to explain the target variation without it.

Key Result

Feature importance in regression is best evaluated by how removing features changes R-squared and Mean Squared Error, revealing which features truly help prediction.

Practice

1. What does feature importance tell us in a regression model?

easy

A. The accuracy of the model on test data

B. The number of layers in the model

C. Which input features most influence the model's predictions

D. The speed at which the model trains

Solution

1. Step 1: Understand feature importance meaning

   Feature importance measures how much each input feature affects the model's output.

2. Step 2: Compare options to definition

   Only Which input features most influence the model's predictions correctly describes this concept as the influence of input features on predictions.

3. Final Answer:

   Which input features most influence the model's predictions -> Option C

4. Quick Check:

   Feature importance = input influence [OK]

Hint: Feature importance ranks inputs by their effect on output [OK]

Common Mistakes:

• Confusing feature importance with model accuracy
• Thinking it shows model speed or size
• Mixing it up with number of model layers

2. Which Python library function is commonly used to get feature importance from a Random Forest regression model?

easy

A. model.score()

B. model.predict()

C. model.fit()

D. model.feature_importances_

Solution

1. Step 1: Recall Random Forest attribute for feature importance

   Random Forest regression models in scikit-learn provide feature importance via the attribute feature_importances_.

2. Step 2: Eliminate other options

   predict() makes predictions, fit() trains the model, and score() evaluates performance, none return feature importance.

3. Final Answer:

   model.feature_importances_ -> Option D

4. Quick Check:

   Feature importance attribute = feature_importances_ [OK]

Hint: Feature importance in Random Forest is model.feature_importances_ [OK]

Common Mistakes:

• Using predict() instead of feature_importances_
• Confusing fit() with importance retrieval
• Trying to get importance from score()

3. Given this code snippet, what will be the output?

from sklearn.ensemble import RandomForestRegressor
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([10, 20, 30, 40])
model = RandomForestRegressor(random_state=0)
model.fit(X, y)
print(model.feature_importances_)

medium

A. [0.3 0.7]

B. [0.7 0.3]

C. [0.5 0.5]

D. [0.0 1.0]

Solution

1. Step 1: Understand the dataset and model

   The features are two columns with increasing values, but the first feature has a slightly stronger linear relationship with the target.

2. Step 2: Interpret feature importance output

   Random Forest will assign higher importance to the feature with stronger influence, which is the first feature here.

3. Final Answer:

   [0.7 0.3] -> Option B

4. Quick Check:

   Feature 1 stronger -> higher importance [OK]

Hint: Feature with stronger effect gets higher importance [OK]

Common Mistakes:

• Assuming equal importance without checking data
• Expecting exact 0.5 values
• Confusing feature importance with coefficients

4. You run this code but get an error:

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor()
print(model.feature_importances_)

What is the problem?

medium

A. The model must be trained with fit() before accessing feature_importances_

B. RandomForestRegressor does not have feature_importances_ attribute

C. You must import feature_importances_ separately

D. feature_importances_ is only for classification models

Solution

1. Step 1: Identify cause of error

   Accessing feature_importances_ before training causes an AttributeError because the model has no learned features yet.

2. Step 2: Correct usage

   Call fit() on training data before accessing feature_importances_.

3. Final Answer:

   The model must be trained with fit() before accessing feature_importances_ -> Option A

4. Quick Check:

   Train model first, then get feature importance [OK]

Hint: Always fit model before checking feature_importances_ [OK]

Common Mistakes:

• Trying to access feature_importances_ before fit()
• Thinking feature_importances_ needs separate import
• Believing it's only for classification

5. You have a regression dataset with 10 features. After training a Random Forest model, you find two features have very low importance scores close to zero. What is the best next step?

hard

A. Remove those low-importance features and retrain the model

B. Increase the number of trees in the forest

C. Add more features similar to the low-importance ones

D. Ignore feature importance and keep all features

Solution

1. Step 1: Interpret low feature importance meaning

   Features with near-zero importance contribute little to predictions and may add noise or complexity.

2. Step 2: Decide action based on importance

   Removing unimportant features simplifies the model and can improve performance and interpretability.

3. Final Answer:

   Remove those low-importance features and retrain the model -> Option A

4. Quick Check:

   Drop low-importance features to simplify model [OK]

Hint: Drop features with near-zero importance to improve model [OK]

Common Mistakes:

• Adding more similar low-importance features
• Ignoring importance and keeping all features
• Only increasing trees without feature selection