Feature importance in regression in ML Python - ML Experiment: Train & Evaluate

Experiment - Feature importance in regression

Problem:You have a dataset with several features to predict house prices using regression. The current model is a simple linear regression.

Current Metrics:Training R2 score: 0.75, Validation R2 score: 0.70

Issue:You want to understand which features are most important for predicting house prices to improve model interpretability and possibly reduce model complexity.

Your Task

Identify the most important features influencing house price predictions using a regression model and visualize their importance.

Use a regression model that supports feature importance extraction.

Do not change the dataset or target variable.

Keep the model simple and interpretable.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score
import matplotlib.pyplot as plt

# Load dataset
housing = fetch_california_housing(as_frame=True)
X = housing.data
y = housing.target

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Random Forest Regressor
model = RandomForestRegressor(random_state=42)
model.fit(X_train, y_train)

# Predict and evaluate
train_pred = model.predict(X_train)
val_pred = model.predict(X_val)
train_r2 = r2_score(y_train, train_pred)
val_r2 = r2_score(y_val, val_pred)

# Extract feature importance
importances = model.feature_importances_
features = X.columns

# Plot feature importance
plt.figure(figsize=(8,5))
plt.barh(features, importances, color='skyblue')
plt.xlabel('Feature Importance')
plt.title('Feature Importance from Random Forest Regressor')
plt.gca().invert_yaxis()
plt.tight_layout()
plt.show()

print(f"Training R2 score: {train_r2:.2f}")
print(f"Validation R2 score: {val_r2:.2f}")

Replaced simple linear regression with Random Forest Regressor to get feature importance.

Added code to extract and plot feature importance for all features.

Evaluated model performance on training and validation sets.

Results Interpretation

Before: Training R2 = 0.75, Validation R2 = 0.70 (Linear Regression)

After: Training R2 = 0.98, Validation R2 = 0.81 (Random Forest Regressor with feature importance)

The new model performs better and provides a clear ranking of feature importance, helping to understand which features most influence house prices.

Using models that provide feature importance helps us understand which inputs matter most. This improves trust and guides feature selection or further model tuning.

Bonus Experiment

Try using permutation feature importance on the trained Random Forest model to confirm the importance ranking.

💡 Hint

Use sklearn.inspection.permutation_importance to measure how shuffling each feature affects model performance.

Practice

1. What does feature importance tell us in a regression model?

easy

A. The accuracy of the model on test data

B. The number of layers in the model

C. Which input features most influence the model's predictions

D. The speed at which the model trains

Solution

1. Step 1: Understand feature importance meaning

   Feature importance measures how much each input feature affects the model's output.

2. Step 2: Compare options to definition

   Only Which input features most influence the model's predictions correctly describes this concept as the influence of input features on predictions.

3. Final Answer:

   Which input features most influence the model's predictions -> Option C

4. Quick Check:

   Feature importance = input influence [OK]

Hint: Feature importance ranks inputs by their effect on output [OK]

Common Mistakes:

• Confusing feature importance with model accuracy
• Thinking it shows model speed or size
• Mixing it up with number of model layers

2. Which Python library function is commonly used to get feature importance from a Random Forest regression model?

easy

A. model.score()

B. model.predict()

C. model.fit()

D. model.feature_importances_

Solution

1. Step 1: Recall Random Forest attribute for feature importance

   Random Forest regression models in scikit-learn provide feature importance via the attribute feature_importances_.

2. Step 2: Eliminate other options

   predict() makes predictions, fit() trains the model, and score() evaluates performance, none return feature importance.

3. Final Answer:

   model.feature_importances_ -> Option D

4. Quick Check:

   Feature importance attribute = feature_importances_ [OK]

Hint: Feature importance in Random Forest is model.feature_importances_ [OK]

Common Mistakes:

• Using predict() instead of feature_importances_
• Confusing fit() with importance retrieval
• Trying to get importance from score()

3. Given this code snippet, what will be the output?

from sklearn.ensemble import RandomForestRegressor
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([10, 20, 30, 40])
model = RandomForestRegressor(random_state=0)
model.fit(X, y)
print(model.feature_importances_)

medium

A. [0.3 0.7]

B. [0.7 0.3]

C. [0.5 0.5]

D. [0.0 1.0]

Solution

1. Step 1: Understand the dataset and model

   The features are two columns with increasing values, but the first feature has a slightly stronger linear relationship with the target.

2. Step 2: Interpret feature importance output

   Random Forest will assign higher importance to the feature with stronger influence, which is the first feature here.

3. Final Answer:

   [0.7 0.3] -> Option B

4. Quick Check:

   Feature 1 stronger -> higher importance [OK]

Hint: Feature with stronger effect gets higher importance [OK]

Common Mistakes:

• Assuming equal importance without checking data
• Expecting exact 0.5 values
• Confusing feature importance with coefficients

4. You run this code but get an error:

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor()
print(model.feature_importances_)

What is the problem?

medium

A. The model must be trained with fit() before accessing feature_importances_

B. RandomForestRegressor does not have feature_importances_ attribute

C. You must import feature_importances_ separately

D. feature_importances_ is only for classification models

Solution

1. Step 1: Identify cause of error

   Accessing feature_importances_ before training causes an AttributeError because the model has no learned features yet.

2. Step 2: Correct usage

   Call fit() on training data before accessing feature_importances_.

3. Final Answer:

   The model must be trained with fit() before accessing feature_importances_ -> Option A

4. Quick Check:

   Train model first, then get feature importance [OK]

Hint: Always fit model before checking feature_importances_ [OK]

Common Mistakes:

• Trying to access feature_importances_ before fit()
• Thinking feature_importances_ needs separate import
• Believing it's only for classification

5. You have a regression dataset with 10 features. After training a Random Forest model, you find two features have very low importance scores close to zero. What is the best next step?

hard

A. Remove those low-importance features and retrain the model

B. Increase the number of trees in the forest

C. Add more features similar to the low-importance ones

D. Ignore feature importance and keep all features

Solution

1. Step 1: Interpret low feature importance meaning

   Features with near-zero importance contribute little to predictions and may add noise or complexity.

2. Step 2: Decide action based on importance

   Removing unimportant features simplifies the model and can improve performance and interpretability.

3. Final Answer:

   Remove those low-importance features and retrain the model -> Option A

4. Quick Check:

   Drop low-importance features to simplify model [OK]

Hint: Drop features with near-zero importance to improve model [OK]

Common Mistakes:

• Adding more similar low-importance features
• Ignoring importance and keeping all features
• Only increasing trees without feature selection