Linear regression with scikit-learn in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Linear regression with scikit-learn

Which metric matters for Linear Regression and WHY

For linear regression, we want to see how close our predicted numbers are to the real numbers. The main metrics are:

• Mean Squared Error (MSE): It shows the average squared difference between predicted and real values. Smaller is better because it means predictions are close.
• Root Mean Squared Error (RMSE): This is the square root of MSE. It is in the same units as the target, making it easier to understand.
• Mean Absolute Error (MAE): It shows the average absolute difference between predicted and real values. It is less sensitive to big errors than MSE.
• R-squared (R²): It tells us how much of the variation in the data is explained by the model. It ranges from negative infinity to 1. Closer to 1 means better fit.

We use these because linear regression predicts continuous numbers, so accuracy or classification metrics don't apply.

Confusion Matrix Equivalent for Regression

Regression does not use a confusion matrix because it predicts numbers, not categories. Instead, we look at error values.

Example errors for 5 samples:

| Sample | Actual | Predicted | Error (Actual - Predicted) |
|--------|--------|-----------|---------------------------|
| 1      | 10     | 9.5       | 0.5                       |
| 2      | 15     | 14.8      | 0.2                       |
| 3      | 7      | 8.1       | -1.1                      |
| 4      | 20     | 19.5      | 0.5                       |
| 5      | 13     | 12.7      | 0.3                       |

From these errors, we calculate MSE, RMSE, MAE.
    

Tradeoffs in Regression Metrics

Choosing the right metric depends on what you care about:

• MSE and RMSE: Penalize big errors more because errors are squared. Good if big mistakes are very bad.
• MAE: Treats all errors equally. Good if you want a balanced view without extra penalty on big errors.
• R²: Shows how well the model explains the data. But a high R² doesn't always mean small errors.

For example, if predicting house prices, a few very wrong predictions can be costly, so RMSE is useful. If you want average error size, MAE is better.

What Good vs Bad Metrics Look Like for Linear Regression

• Good:
  • Low MSE and RMSE close to zero (e.g., RMSE = 2 for house prices in thousands)
  • Low MAE (small average error, e.g., 1.5)
  • High R² close to 1 (e.g., 0.9 means 90% of variation explained)
• Bad:
  • High MSE and RMSE (large errors, e.g., RMSE = 20)
  • High MAE (large average error, e.g., 15)
  • Low R² close to 0 or negative (model explains little or no variation)

Common Pitfalls in Regression Metrics

• Ignoring scale: MSE and RMSE depend on the scale of the target. Comparing errors across different datasets without scaling can mislead.
• Overfitting: Very low training error but high test error means the model learned noise, not patterns.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• R² misuse: R² can be negative if the model is worse than just predicting the mean. Don't assume high R² always means good predictions.

Self Check

Your linear regression model has an R² of 0.98 on training data but 0.5 on test data. Is it good?

Answer: No, this shows overfitting. The model fits training data very well but does not generalize to new data. You should improve the model or get more data.

Key Result

For linear regression, low RMSE and high R² show good predictions; watch out for overfitting and scale effects.

Practice

1. What is the main goal of linear regression in machine learning?

easy

A. To find a line that predicts a continuous value from input data

B. To classify data into categories

C. To cluster data points into groups

D. To reduce the number of features in data

Solution

1. Step 1: Understand the purpose of linear regression

   Linear regression aims to model the relationship between input variables and a continuous output by fitting a line.

2. Step 2: Compare with other tasks

   Classification assigns categories, clustering groups data, and feature reduction simplifies data, which are different tasks.

3. Final Answer:

   To find a line that predicts a continuous value from input data -> Option A

4. Quick Check:

   Linear regression predicts continuous values [OK]

Hint: Linear regression predicts continuous numbers, not categories [OK]

Common Mistakes:

• Confusing regression with classification
• Thinking linear regression clusters data
• Believing it reduces features

2. Which of the following is the correct way to import the LinearRegression class from scikit-learn?

easy

A. from sklearn.linear_model import LinearRegression

B. import LinearRegression from sklearn.linear_model

C. from sklearn import LinearRegression

D. import linear_model.LinearRegression

Solution

1. Step 1: Recall correct Python import syntax

   Python imports classes using 'from module import ClassName' syntax.

2. Step 2: Match with scikit-learn structure

   LinearRegression is inside sklearn.linear_model, so correct import is from sklearn.linear_model import LinearRegression.

3. Final Answer:

   from sklearn.linear_model import LinearRegression -> Option A

4. Quick Check:

   Correct import syntax = from sklearn.linear_model import LinearRegression [OK]

Hint: Use 'from sklearn.linear_model import LinearRegression' [OK]

Common Mistakes:

• Using wrong import order
• Trying to import directly from sklearn
• Using incorrect module names

3. What will be the output of the following code snippet?

from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([2, 4, 6, 8, 10])
model = LinearRegression()
model.fit(X, y)
print(round(model.coef_[0], 1))

medium

A. 0.5

B. 2.0

C. 10.0

D. 1.0

Solution

1. Step 1: Understand the data relationship

   X values are 1 to 5, y values are exactly double X (2,4,6,8,10), so slope should be 2.

2. Step 2: Check model coefficient

   LinearRegression fits y = coef * X + intercept. coef_ will be close to 2.0 here.

3. Final Answer:

   2.0 -> Option B

4. Quick Check:

   Slope of y=2x is 2.0 [OK]

Hint: Slope equals change in y over change in x [OK]

Common Mistakes:

• Confusing coefficient with intercept
• Not rounding output
• Assuming coefficient is 1 or 10

4. Identify the error in this code that tries to train a linear regression model:

from sklearn.linear_model import LinearRegression
X = [[1], [2], [3]]
y = [2, 4, 6]
model = LinearRegression()
model.fit(X, y)
print(model.predict([4]))

medium

A. Passing a list instead of numpy array to fit

B. LinearRegression class is not instantiated correctly

C. Missing import for numpy

D. Predict input shape is incorrect, should be 2D array

Solution

1. Step 1: Check input shapes for fit and predict

   fit accepts list of lists for X, so no error there. But predict expects 2D array, [4] is 1D.

2. Step 2: Fix predict input shape

   Predict should get [[4]] (2D) not [4] (1D) to avoid shape error.

3. Final Answer:

   Predict input shape is incorrect, should be 2D array -> Option D

4. Quick Check:

   Predict input must be 2D array [OK]

Hint: Always pass 2D array to predict, e.g., [[value]] [OK]

Common Mistakes:

• Passing 1D array to predict
• Assuming fit needs numpy arrays
• Wrong class instantiation

5. You trained a linear regression model with scikit-learn and got an R² score of 0.85 on test data. What does this score mean?

hard

A. The model has 85% chance to predict correctly

B. The model's predictions are 85% accurate

C. The model explains 85% of the variance in the target variable

D. The model's error rate is 15% on test data

Solution

1. Step 1: Understand R² score meaning

   R² (coefficient of determination) measures how well the model explains variance in the target.

2. Step 2: Interpret 0.85 R²

   A score of 0.85 means 85% of the variance in the target is explained by the model.

3. Final Answer:

   The model explains 85% of the variance in the target variable -> Option C

4. Quick Check:

   R² = variance explained, not accuracy or error rate [OK]

Hint: R² shows variance explained, not accuracy percentage [OK]

Common Mistakes:

• Confusing R² with accuracy
• Thinking R² is probability
• Assuming R² is error rate