Polynomial regression in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Polynomial regression

Which metric matters for Polynomial Regression and WHY

Polynomial regression predicts continuous values. So, we want metrics that show how close predictions are to actual numbers.

Mean Squared Error (MSE) is common. It measures average squared difference between predicted and actual values. Smaller MSE means better fit.

Root Mean Squared Error (RMSE) is the square root of MSE. It is in the same units as the target, making it easier to understand.

R-squared (R²) shows how much of the data's variation the model explains. Closer to 1 means better fit.

We use these because polynomial regression is about fitting curves to data points, so measuring prediction errors and explained variance is key.

Confusion Matrix or Equivalent Visualization

Polynomial regression is for continuous output, so confusion matrix (used for classification) does not apply.

Instead, we look at error values. For example, given actual and predicted values:

    Actual:    [3, 5, 7, 9]
    Predicted: [2.8, 5.1, 6.9, 9.2]

    Errors:    [-0.2, 0.1, -0.1, 0.2]
    Squared Errors: [0.04, 0.01, 0.01, 0.04]
    MSE = (0.04 + 0.01 + 0.01 + 0.04) / 4 = 0.025
    RMSE = sqrt(0.025) = 0.1581
    

This shows how close predictions are to actual values.

Tradeoff: Underfitting vs Overfitting

Polynomial regression degree controls model complexity.

Low degree (e.g., 1 or 2): Model may be too simple. It misses patterns (underfitting). Errors stay high on training and new data.

High degree (e.g., 10 or more): Model fits training data very closely, even noise. It may fail on new data (overfitting). Training error is low but test error is high.

Good model balances complexity to minimize error on new data.

What "Good" vs "Bad" Metric Values Look Like

Good polynomial regression:

• Low MSE and RMSE on both training and test data.
• R² close to 1 (e.g., 0.8 or higher) on test data.
• Errors are small and consistent.

Bad polynomial regression:

• High MSE and RMSE on test data (even if training error is low).
• R² near 0 or negative on test data.
• Large differences between training and test errors (sign of overfitting).
• Very high errors on both training and test (sign of underfitting).

Common Pitfalls in Polynomial Regression Metrics

• Ignoring overfitting: Low training error but high test error means model memorizes noise.
• Using too high degree: Complex models fit noise, not true pattern.
• Relying only on training error: It can be misleading; always check test error.
• Not scaling features: Polynomial terms can grow large, causing instability.
• Data leakage: Using test data in training inflates metrics falsely.

Self Check

Your polynomial regression model has 98% accuracy but 12% recall on fraud. Is it good?

Note: Accuracy and recall are classification metrics, not for regression. This shows a misunderstanding.

For polynomial regression, focus on MSE, RMSE, and R². Using classification metrics here is a pitfall.

So, no, this model evaluation is not correct for polynomial regression.

Key Result

For polynomial regression, low MSE/RMSE and high R² on test data show good model fit balancing underfitting and overfitting.

Practice

1. What is the main purpose of polynomial regression in machine learning?

easy

A. To perform clustering on unlabeled data

B. To reduce the number of features in the dataset

C. To classify data into categories

D. To model non-linear relationships by adding powers of input features

Solution

1. Step 1: Understand the goal of polynomial regression

   Polynomial regression extends linear regression by adding powers of input features to capture curves in data.

2. Step 2: Compare options with this goal

   Only To model non-linear relationships by adding powers of input features describes modeling non-linear relationships by adding powers of features, which matches polynomial regression.

3. Final Answer:

   To model non-linear relationships by adding powers of input features -> Option D

4. Quick Check:

   Polynomial regression = model curves [OK]

Hint: Polynomial regression fits curves by adding powers of inputs [OK]

Common Mistakes:

• Confusing polynomial regression with classification
• Thinking it reduces features instead of expanding them
• Mixing it up with clustering methods

2. Which Python code snippet correctly creates polynomial features of degree 3 using scikit-learn?

easy

A. from sklearn.linear_model import PolynomialFeatures
poly = PolynomialFeatures(3)
X_poly = poly.transform(X)

B. from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=3)
X_poly = poly.fit_transform(X)

C. from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit(X)

D. from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=3)
X_poly = poly.transform(X)

Solution

1. Step 1: Check import and class usage

   PolynomialFeatures is in sklearn.preprocessing, so import must be from there.

2. Step 2: Verify method calls and degree

   Degree=3 is correct. fit_transform(X) creates polynomial features properly. from sklearn.preprocessing import PolynomialFeatures poly = PolynomialFeatures(degree=3) X_poly = poly.fit_transform(X) matches all these.

3. Final Answer:

   from sklearn.preprocessing import PolynomialFeatures poly = PolynomialFeatures(degree=3) X_poly = poly.fit_transform(X) -> Option B

4. Quick Check:

   Correct import + degree + fit_transform = A [OK]

Hint: Use fit_transform with PolynomialFeatures from sklearn.preprocessing [OK]

Common Mistakes:

• Importing PolynomialFeatures from wrong module
• Using transform without fit first
• Setting wrong degree value

3. Given this code, what will be the shape of X_poly if X has shape (5, 1) and degree=2?

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

medium

A. (5, 1)

B. (5, 2)

C. (5, 3)

D. (5, 4)

Solution

1. Step 1: Understand PolynomialFeatures output shape

   For degree=2 and one feature, output includes [1, x, x^2], so 3 features total.

2. Step 2: Calculate shape based on input rows and new features

   Input has 5 rows, so output shape is (5, 3).

3. Final Answer:

   (5, 3) -> Option C

4. Quick Check:

   Degree 2 with 1 feature = 3 columns [OK]

Hint: Output columns = combinations with degree + 1 for bias [OK]

Common Mistakes:

• Forgetting the bias (constant 1) column
• Assuming output shape equals input shape
• Confusing degree with number of features

4. Identify the error in this polynomial regression code snippet:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
poly = PolynomialFeatures(degree=2)
X_poly = poly.transform(X)
model = LinearRegression()
model.fit(X_poly, y)

medium

A. Using transform before fitting PolynomialFeatures

B. LinearRegression cannot be used with polynomial features

C. Missing import for LinearRegression

D. Degree should be 1 for polynomial regression

Solution

1. Step 1: Check usage of PolynomialFeatures methods

   PolynomialFeatures must be fitted before transform; fit_transform fits and transforms in one step.

2. Step 2: Identify the error in code

   Code calls poly.transform(X) without fitting first, causing error. Correct is poly.fit_transform(X).

3. Final Answer:

   Using transform before fitting PolynomialFeatures -> Option A

4. Quick Check:

   fit_transform needed before transform [OK]

Hint: Always fit PolynomialFeatures before transform or use fit_transform [OK]

Common Mistakes:

• Calling transform without fit first
• Thinking LinearRegression can't handle polynomial features
• Ignoring method order in preprocessing

5. You have data with a clear curve pattern. You fit a polynomial regression model with degree 5 and get very low training error but high error on new data. What is the best next step?

hard

A. Reduce the polynomial degree to avoid overfitting

B. Increase the polynomial degree to 10 for better fit

C. Remove polynomial features and use linear regression only

D. Add more polynomial features of degree 6

Solution

1. Step 1: Understand the problem of low training but high test error

   This indicates overfitting: model fits training data too closely but fails to generalize.

2. Step 2: Choose a solution to reduce overfitting

   Reducing polynomial degree simplifies the model, helping it generalize better on new data.

3. Final Answer:

   Reduce the polynomial degree to avoid overfitting -> Option A

4. Quick Check:

   Overfitting? Lower degree polynomial [OK]

Hint: Lower degree to reduce overfitting when test error is high [OK]

Common Mistakes:

• Increasing degree worsens overfitting
• Removing polynomial features loses curve fit
• Adding higher degree features increases complexity