Multiple linear regression in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Multiple linear regression

Which metric matters for Multiple linear regression and WHY

For multiple linear regression, we want to know how well the model predicts continuous values. The key metrics are:

• Mean Squared Error (MSE): Measures average squared difference between predicted and actual values. Lower is better.
• Root Mean Squared Error (RMSE): Square root of MSE, in the same units as the target. Easier to interpret.
• R-squared (R²): Shows how much of the variation in the data the model explains. Closer to 1 means better fit.

These metrics help us understand prediction accuracy and model usefulness.

Confusion matrix or equivalent visualization

Multiple linear regression predicts numbers, not categories, so confusion matrix does not apply.

Instead, imagine a scatter plot of actual vs predicted values:

Actual Values:    3.0, 5.0, 7.0, 9.0
Predicted Values: 2.8, 5.1, 6.9, 9.2

Points close to the diagonal line mean good predictions.

Tradeoff: Bias vs Variance in Multiple linear regression

In multiple linear regression, we balance two errors:

• Bias: When the model is too simple, it misses patterns (underfitting). This causes high error on training and test data.
• Variance: When the model is too complex, it fits noise in training data (overfitting). This causes low training error but high test error.

Good models find a balance, minimizing overall error.

What "good" vs "bad" metric values look like for Multiple linear regression

• Good: RMSE is low compared to the range of target values, and R² is close to 1 (e.g., 0.8 or higher).
• Bad: RMSE is large, close to or larger than the average target value, and R² is near 0 or negative, meaning the model predicts worse than a simple average.

Example:

Target range: 0 to 100
Good RMSE: 5 (small error)
Bad RMSE: 50 (large error)
Good R²: 0.85
Bad R²: 0.1 or negative

Common pitfalls in metrics for Multiple linear regression

• Ignoring scale: Using MSE without considering target scale can mislead. RMSE is easier to interpret.
• Overfitting: Very low training error but high test error means model memorizes noise.
• Data leakage: Using future or test data features in training inflates metrics falsely.
• R² misuse: R² can be negative if model is worse than mean prediction; always check alongside error metrics.

Self-check question

Your multiple linear regression model has an RMSE of 20 on test data, but the target values range from 0 to 100. The R² is 0.15. Is this model good? Why or why not?

Answer: No, the model is not good. An RMSE of 20 is quite large compared to the target range, meaning predictions are often off by 20 units. An R² of 0.15 means the model explains only 15% of the variation, which is low. The model likely needs improvement.

Key Result

For multiple linear regression, low RMSE and high R² indicate good prediction accuracy and model fit.

Practice

1. What does multiple linear regression do in simple terms?

easy

A. It predicts a result using several input features combined.

B. It groups data into clusters based on similarity.

C. It reduces the number of features by combining them.

D. It finds the shortest path between two points.

Solution

1. Step 1: Understand the purpose of multiple linear regression

   Multiple linear regression uses multiple inputs to predict a single output by finding the best weights for each input.

2. Step 2: Compare options with this understanding

   Only It predicts a result using several input features combined. correctly describes predicting a result using several inputs combined.

3. Final Answer:

   It predicts a result using several input features combined. -> Option A

4. Quick Check:

   Multiple inputs predict output = A [OK]

Hint: Remember: multiple inputs, one output prediction [OK]

Common Mistakes:

• Confusing regression with clustering
• Thinking it reduces features instead of weighting them
• Mixing it up with pathfinding algorithms

2. Which of the following is the correct formula for multiple linear regression prediction?

easy

A. y = w0 + w1*x1 + w2*x2 + ... + wn*xn

B. y = w0 * x1 + w1 * x2 + ... + wn * xn

C. y = w0 / x1 + w1 / x2 + ... + wn / xn

D. y = w0 - w1*x1 - w2*x2 - ... - wn*xn

Solution

1. Step 1: Recall the multiple linear regression formula

   The prediction is a sum of a bias term plus each input multiplied by its weight: y = w0 + w1*x1 + w2*x2 + ... + wn*xn.

2. Step 2: Check each option against the formula

   Only y = w0 + w1*x1 + w2*x2 + ... + wn*xn matches the correct formula with addition and multiplication.

3. Final Answer:

   y = w0 + w1*x1 + w2*x2 + ... + wn*xn -> Option A

4. Quick Check:

   Sum of weighted inputs plus bias = B [OK]

Hint: Sum weighted inputs plus bias for prediction [OK]

Common Mistakes:

• Using multiplication instead of addition between terms
• Dividing weights by inputs
• Subtracting weights instead of adding

3. Given weights w0=2, w1=3, w2=-1 and inputs x1=4, x2=5, what is the predicted value y using multiple linear regression?

medium

A. 9.5

B. 9

C. 5

D. 10

Solution

1. Step 1: Apply the formula y = w0 + w1*x1 + w2*x2

   Substitute values: y = 2 + 3*4 + (-1)*5.

2. Step 2: Calculate the prediction

   Calculate: 3*4 = 12, (-1)*5 = -5, sum = 2 + 12 - 5 = 9.

3. Step 3: Recheck calculation

   Sum is 9, but 10 says 10, so re-check: 2 + 12 - 5 = 9, so correct answer is 9.

4. Final Answer:

   9 -> Option B

5. Quick Check:

   2 + 12 - 5 = 9 [OK]

Hint: Multiply weights by inputs, then add bias [OK]

Common Mistakes:

• Adding weights instead of multiplying by inputs
• Forgetting to add bias term w0
• Mixing up signs of weights

4. Identify the error in this Python code for multiple linear regression prediction:

def predict(x1, x2, w0, w1, w2):
    return w0 + w1 * x1 + w2 + x2

print(predict(3, 4, 1, 2, 3))

medium

A. The print statement syntax is incorrect.

B. The function is missing a return statement.

C. The function parameters are in wrong order.

D. The term w2 + x2 should be w2 * x2 for correct multiplication.

Solution

1. Step 1: Analyze the formula inside the function

   The formula should multiply each input by its weight: w0 + w1*x1 + w2*x2.

2. Step 2: Identify the mistake in the code

   The code uses w2 + x2 instead of w2 * x2, which adds instead of multiplies.

3. Final Answer:

   The term w2 + x2 should be w2 * x2 for correct multiplication. -> Option D

4. Quick Check:

   Multiplication needed, not addition = A [OK]

Hint: Check all input-weight pairs use multiplication [OK]

Common Mistakes:

• Using + instead of * between weight and input
• Forgetting return statement (not here though)
• Mixing parameter order
• Incorrect print syntax

5. You have a dataset with features: hours studied (x1), hours slept (x2), and exam score (y). You fit a multiple linear regression model and get weights: w0=5, w1=4, w2=2. What does the weight w1=4 tell you about hours studied?

hard

A. Hours slept affects exam score more than hours studied.

B. Hours studied has no effect on exam score.

C. For each extra hour studied, exam score increases by 4 points, holding sleep constant.

D. Exam score decreases by 4 points for each hour studied.

Solution

1. Step 1: Understand the meaning of weights in multiple linear regression

   Each weight shows how much the output changes when that input increases by one unit, keeping others fixed.

2. Step 2: Interpret w1=4 for hours studied

   A weight of 4 means exam score increases by 4 points for each additional hour studied, assuming hours slept stays the same.

3. Final Answer:

   For each extra hour studied, exam score increases by 4 points, holding sleep constant. -> Option C

4. Quick Check:

   Weight shows effect per unit input = D [OK]

Hint: Weight = change in output per unit input [OK]

Common Mistakes:

• Thinking weight means no effect
• Confusing positive and negative effect
• Mixing effects of different features