Polynomial regression pipeline in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Polynomial regression pipeline

Which metric matters for Polynomial Regression and WHY

For polynomial regression, we want to measure how close our predicted values are to the actual values. The key metric is Mean Squared Error (MSE) or Root Mean Squared Error (RMSE). These metrics tell us the average squared difference between predicted and true values. Lower values mean better predictions.

We also look at R-squared (R²), which shows how much of the variation in the data our model explains. R² ranges from 0 to 1, where 1 means perfect prediction.

Confusion Matrix or Equivalent Visualization

Polynomial regression is a regression task, not classification, so we do not use a confusion matrix. Instead, we visualize results with a plot:

    Actual values:    *   *     *  *    *
    Predicted curve:  ---\___/---\___/---
    
    The closer the curve is to the stars, the better the model.

This visual helps us see if the model fits the data well or misses important patterns.

Tradeoff: Underfitting vs Overfitting

Polynomial regression degree controls model complexity:

• Low degree (e.g., 1 or 2): Model is simple and may miss patterns (underfitting). MSE will be high.
• High degree (e.g., 10+): Model fits training data very closely but may fail on new data (overfitting). Training MSE is low but test MSE is high.

We want to find a degree that balances this tradeoff, giving low error on both training and new data.

What "Good" vs "Bad" Metric Values Look Like

Good polynomial regression results:

• Low MSE/RMSE: Close to 0, meaning predictions are near actual values.
• High R²: Close to 1, meaning model explains most data variation.

Bad results:

• High MSE/RMSE: Large errors, model predictions far from actual.
• Low or negative R²: Model explains little or no variation, worse than just guessing the average.

Common Pitfalls in Polynomial Regression Metrics

• Overfitting: Very low training error but high test error means model memorizes noise, not true pattern.
• Ignoring test data: Only checking training error can mislead about real performance.
• Using R² alone: High R² on training data can hide overfitting; always check test error.
• Data leakage: Using future or test data in training inflates metrics falsely.

Self-Check Question

Your polynomial regression model has a training RMSE of 0.5 but a test RMSE of 5.0. Is this good? Why or why not?

Answer: This is not good. The large difference means the model fits training data well but performs poorly on new data. This is overfitting. You should try a simpler model or use regularization.

Key Result

Mean Squared Error and R-squared are key metrics to evaluate polynomial regression, balancing fit quality and generalization.

Practice

1.

What is the main purpose of using polynomial regression instead of simple linear regression?

easy

A. To fit curved relationships between variables

B. To reduce the number of features

C. To speed up training time

D. To handle missing data automatically

Solution

1. Step 1: Understand linear regression limitation

   Linear regression fits straight lines, which cannot capture curves in data.

2. Step 2: Role of polynomial regression

   Polynomial regression fits curved lines by adding powers of features, capturing non-linear patterns.

3. Final Answer:

   To fit curved relationships between variables -> Option A

4. Quick Check:

   Polynomial regression = curved fit [OK]

Hint: Polynomial regression fits curves, not just straight lines [OK]

Common Mistakes:

• Thinking polynomial regression reduces features
• Assuming it speeds up training
• Believing it handles missing data automatically

2.

Which of the following is the correct way to create a polynomial regression pipeline in Python using sklearn?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

pipeline = Pipeline([
    ('poly', PolynomialFeatures(degree=2)),
    ('linear', LinearRegression())
])

easy

A. pipeline = Pipeline([('poly', PolynomialFeatures(degree=2)), ('linear', LinearRegression())])

B. pipeline = Pipeline([('linear', LinearRegression()), ('poly', PolynomialFeatures(degree=2))])

C. pipeline = Pipeline([('poly', LinearRegression()), ('linear', PolynomialFeatures(degree=2))])

D. pipeline = Pipeline([('poly', PolynomialFeatures()), ('linear', LinearRegression(degree=2))])

Solution

1. Step 1: Order of pipeline steps

   PolynomialFeatures must come before LinearRegression to transform data first.

2. Step 2: Correct usage of classes and parameters

   PolynomialFeatures takes degree parameter; LinearRegression does not take degree.

3. Final Answer:

   pipeline = Pipeline([('poly', PolynomialFeatures(degree=2)), ('linear', LinearRegression())]) -> Option A

4. Quick Check:

   PolynomialFeatures before LinearRegression [OK]

Hint: Put PolynomialFeatures before LinearRegression in pipeline [OK]

Common Mistakes:

• Swapping order of pipeline steps
• Passing degree to LinearRegression
• Omitting degree in PolynomialFeatures

3.

Given the following code, what will print(y_pred) output?

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3]])
y = np.array([1, 4, 9])

pipeline = Pipeline([
    ('poly', PolynomialFeatures(degree=2)),
    ('linear', LinearRegression())
])
pipeline.fit(X, y)
y_pred = pipeline.predict(np.array([[4]]))
print(np.round(y_pred, 2))

medium

A. [10.0]

B. [8.0]

C. [4.0]

D. [16.0]

Solution

1. Step 1: Understand data and model

   X = [[1],[2],[3]] with y = [1,4,9] fits y = x^2 perfectly.

2. Step 2: Predict for X=4 using polynomial degree 2

   Model learns y = x^2, so prediction at 4 is 4^2 = 16.

3. Final Answer:

   [16.0] -> Option D

4. Quick Check:

   4 squared = 16 [OK]

Hint: Polynomial degree 2 fits squares; predict 4^2 = 16 [OK]

Common Mistakes:

• Ignoring polynomial transformation
• Predicting linear value instead of squared
• Rounding errors without np.round

4.

Identify the error in this polynomial regression pipeline code:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

pipeline = Pipeline([
    ('linear', LinearRegression()),
    ('poly', PolynomialFeatures(degree=3))
])

pipeline.fit(X_train, y_train)

medium

A. LinearRegression should not be used in pipeline

B. The order of pipeline steps is incorrect

C. PolynomialFeatures degree must be 2, not 3

D. Missing import for X_train and y_train

Solution

1. Step 1: Check pipeline step order

   PolynomialFeatures must come before LinearRegression to transform data first.

2. Step 2: Confirm degree and imports

   Degree 3 is valid; imports for data are assumed outside snippet.

3. Final Answer:

   The order of pipeline steps is incorrect -> Option B

4. Quick Check:

   PolynomialFeatures before LinearRegression [OK]

Hint: PolynomialFeatures must be first in pipeline [OK]

Common Mistakes:

• Swapping order of steps
• Thinking degree must be 2
• Confusing missing data imports with pipeline error

5.

You want to model a dataset with a complex curve. You try polynomial regression with degree=2 but the fit is poor. What is the best next step?

hard

A. Remove polynomial features and use linear regression only

B. Decrease the polynomial degree to avoid overfitting

C. Increase the polynomial degree to capture more complexity

D. Use degree=2 but reduce training data size

Solution

1. Step 1: Understand model complexity and fit

   Degree 2 polynomial may be too simple for complex curves, causing poor fit.

2. Step 2: Adjust polynomial degree

   Increasing degree allows model to fit more complex patterns, improving fit quality.

3. Final Answer:

   Increase the polynomial degree to capture more complexity -> Option C

4. Quick Check:

   Higher degree = better complex fit [OK]

Hint: Raise degree to fit complex curves better [OK]

Common Mistakes:

• Lowering degree when fit is poor
• Removing polynomial features unnecessarily
• Reducing data size instead of model complexity