Residual analysis in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Residual analysis

Which metric matters for Residual Analysis and WHY

Residual analysis focuses on the differences between what the model predicts and the actual values. These differences are called residuals. The key metric is the residual value itself, often summarized by Mean Squared Error (MSE) or Root Mean Squared Error (RMSE). These metrics tell us how far off the predictions are on average.

We use residual analysis to check if the model fits well and if errors are random. If residuals show patterns, the model might be missing something important.

Confusion Matrix or Equivalent Visualization

Residual analysis does not use a confusion matrix because it is for regression, not classification. Instead, we look at a residual plot. This plot shows residuals (errors) on the vertical axis and predicted or actual values on the horizontal axis.

Predicted Value --->
|
|    *     *    *    *    *
|  *   *   *   *   *   *  
| * * * * * * * * * * * * 
|_________________________

Residual (Error) on vertical axis

Ideally, points scatter randomly around zero with no clear pattern.
    

Tradeoff: Residual Size vs Model Complexity

When building a model, smaller residuals mean better fit. But if residuals are too small, the model might be too complex and just memorizing data (overfitting).

Simple models may have larger residuals but generalize better. Complex models reduce residuals on training data but may fail on new data.

Residual analysis helps find a balance: residuals should be small and show no pattern, indicating the model captures the main trends without overfitting.

What "Good" vs "Bad" Residuals Look Like

Good residuals:

• Scatter randomly around zero
• Have similar spread across all predicted values
• Have small average size (low MSE or RMSE)

Bad residuals:

• Show clear patterns or curves (indicating missing relationships)
• Increase or decrease as predicted values change (heteroscedasticity)
• Are large on average (high MSE or RMSE)

Common Pitfalls in Residual Analysis

• Ignoring patterns: Residuals with patterns mean the model misses key trends.
• Assuming small residuals always mean good model: Overfitting can cause very small residuals on training data but poor performance on new data.
• Not checking residual spread: Residuals should have constant variance; if not, model assumptions are violated.
• Data leakage: If future information leaks into training, residuals may look artificially small.

Self-Check Question

Your regression model has a low average residual error (small MSE), but the residual plot shows a clear curve pattern. Is this model good? Why or why not?

Answer: No, the model is not good. The curve pattern in residuals means the model misses some relationship in the data. Even if average error is low, the model does not capture all trends and may give biased predictions.

Key Result

Residual analysis uses residual values and their patterns to check model fit and detect missing trends.

Practice

1. What does a residual represent in machine learning?

easy

A. The difference between the actual and predicted values

B. The predicted value itself

C. The average of all predicted values

D. The sum of all actual values

Solution

1. Step 1: Understand residual definition

   A residual is the error between what the model predicts and what actually happened.

2. Step 2: Match with options

   The difference between the actual and predicted values correctly states this difference, while others describe unrelated concepts.

3. Final Answer:

   The difference between the actual and predicted values -> Option A

4. Quick Check:

   Residual = Actual - Predicted [OK]

Hint: Residuals = Actual minus Predicted values [OK]

Common Mistakes:

• Confusing residual with predicted value
• Thinking residual is the average error
• Mixing residual with total sum of values

2. Which of the following is the correct formula to calculate a residual for a data point?

easy

A. Residual = Actual value + Predicted value

B. Residual = Predicted value - Actual value

C. Residual = Actual value - Predicted value

D. Residual = Predicted value / Actual value

Solution

1. Step 1: Recall residual calculation formula

   Residual is calculated as Actual value minus Predicted value.

2. Step 2: Identify correct option

   Residual = Actual value - Predicted value matches the formula exactly; others are incorrect operations.

3. Final Answer:

   Residual = Actual value - Predicted value -> Option C

4. Quick Check:

   Residual = Actual - Predicted [OK]

Hint: Subtract predicted from actual to get residual [OK]

Common Mistakes:

• Reversing the subtraction order
• Adding values instead of subtracting
• Dividing values instead of subtracting

3. Given the actual values [3, 5, 7] and predicted values [2, 5, 8], what is the residual for the third data point?

medium

A. 1

B. -1

C. 0

D. 2

Solution

1. Step 1: Identify actual and predicted values for third point

   Actual = 7, Predicted = 8 for the third data point.

2. Step 2: Calculate residual

   Residual = Actual - Predicted = 7 - 8 = -1.

3. Final Answer:

   -1 -> Option B

4. Quick Check:

   7 - 8 = -1 [OK]

Hint: Subtract predicted from actual for the point [OK]

Common Mistakes:

• Mixing up actual and predicted values
• Calculating predicted minus actual
• Using wrong index for data point

4. You have a residuals list: [2, -3, 4, -1]. Which code snippet correctly calculates the mean squared error (MSE)?

medium

A. mse = sum([r**2 for r in residuals]) / len(residuals)

B. mse = sum([abs(r) for r in residuals]) / len(residuals)

C. mse = max(residuals) - min(residuals)

D. mse = sum(residuals) / len(residuals)

Solution

1. Step 1: Understand MSE formula

   MSE is the average of squared residuals, so square each residual and average them.

2. Step 2: Match code to formula

   mse = sum([r**2 for r in residuals]) / len(residuals) squares each residual and divides by count, matching MSE calculation.

3. Final Answer:

   mse = sum([r**2 for r in residuals]) / len(residuals) -> Option A

4. Quick Check:

   MSE = average of squared residuals [OK]

Hint: Square residuals then average for MSE [OK]

Common Mistakes:

• Using sum instead of sum of squares
• Calculating mean absolute error instead
• Using max-min difference instead of average

5. A model's residual plot shows a clear curve pattern instead of random scatter. What does this indicate about the model?

hard

A. The model fits the data perfectly

B. The model has no bias

C. The residuals are normally distributed

D. The model misses a non-linear pattern in data

Solution

1. Step 1: Interpret residual plot pattern

   A curve pattern in residuals suggests systematic error, meaning the model misses some pattern.

2. Step 2: Identify what pattern means

   This usually means the model is too simple and misses non-linear relationships.

3. Final Answer:

   The model misses a non-linear pattern in data -> Option D

4. Quick Check:

   Curved residuals = missed non-linear pattern [OK]

Hint: Curved residuals mean model misses pattern [OK]

Common Mistakes:

• Assuming perfect fit from residual pattern
• Confusing residual distribution with residual pattern
• Ignoring bias presence in model