Feature importance in regression in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is feature importance in regression?

Feature importance in regression tells us how much each input feature (variable) helps the model predict the target value. It shows which features matter most.

beginner

Name one simple method to calculate feature importance in regression models.

One simple method is to look at the coefficients of a linear regression model. Larger absolute values mean the feature has more influence on the prediction.

intermediate

How does permutation importance work for regression models?

Permutation importance measures feature importance by randomly shuffling one feature's values and seeing how much the model's prediction error increases. A big increase means the feature is important.

intermediate

Why might feature importance from tree-based models be more reliable than from linear regression?

Tree-based models capture complex relationships and interactions between features, so their importance scores reflect more realistic influence, while linear regression assumes simple linear effects.

advanced

What is a limitation of using feature importance scores directly for feature selection?

Feature importance scores can be biased if features are correlated or if the model is complex. Removing features based only on importance might hurt model performance.

Which method uses model coefficients to determine feature importance in regression?

A Linear regression coefficients

B Permutation importance

C SHAP values

D PCA

What does a large increase in error after shuffling a feature indicate in permutation importance?

A The feature is redundant

B The feature is not important

C The model is overfitting

D The feature is important

Which model type often provides feature importance by measuring splits and gains?

A Decision trees

B K-means clustering

C Linear regression

D Naive Bayes

Why can correlated features cause problems in feature importance?

A They always increase importance scores

B They can share importance, making scores unreliable

C They reduce model accuracy

D They are ignored by models

Which of these is NOT a direct way to get feature importance in regression?

A Model coefficients

B Permutation importance

C Feature scaling

D Tree-based importance

Explain how permutation importance helps identify important features in regression models.

Describe why feature importance from tree-based regression models might be more informative than from linear regression.