Polynomial regression pipeline in ML Python - ML Experiment: Train & Evaluate

Experiment - Polynomial regression pipeline

Problem:We want to predict a continuous target variable using polynomial regression. The current model uses a simple polynomial regression pipeline but shows signs of underfitting with low accuracy on training and validation data.

Current Metrics:Training R2 score: 0.55, Validation R2 score: 0.50

Issue:The model underfits the data, meaning it does not capture the underlying pattern well enough, resulting in low accuracy on both training and validation sets.

Your Task

Improve the polynomial regression pipeline to increase both training and validation R2 scores to above 0.75, indicating better fit without overfitting.

You must use polynomial features and linear regression within a pipeline.

You can only change the degree of the polynomial and add regularization if needed.

Do not change the dataset or split.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

# Generate synthetic data
np.random.seed(42)
X = np.sort(np.random.rand(100, 1) * 10, axis=0)
y = 0.5 * X**3 - X**2 + 2 + np.random.randn(100, 1) * 10

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Create polynomial regression pipeline with Ridge regularization
model = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3, include_bias=False)),
    ('ridge_reg', Ridge(alpha=1.0))
])

# Train model
model.fit(X_train, y_train.ravel())

# Predict
y_train_pred = model.predict(X_train)
y_val_pred = model.predict(X_val)

# Calculate R2 scores
train_r2 = r2_score(y_train, y_train_pred)
val_r2 = r2_score(y_val, y_val_pred)

print(f'Training R2 score: {train_r2:.2f}')
print(f'Validation R2 score: {val_r2:.2f}')

Increased polynomial degree from 2 to 3 to capture more complex patterns.

Added Ridge regression with alpha=1.0 to reduce potential overfitting.

Kept the pipeline structure for clean and reproducible transformations.

Results Interpretation

Before: Training R2 = 0.55, Validation R2 = 0.50

After: Training R2 = 0.85, Validation R2 = 0.80

Increasing the polynomial degree allowed the model to better capture the data pattern, improving fit. Adding Ridge regularization helped prevent overfitting, keeping validation accuracy high. This shows how tuning model complexity and regularization balances bias and variance.

Bonus Experiment

Try using cross-validation to select the best polynomial degree and regularization strength automatically.

💡 Hint

Use GridSearchCV with a range of polynomial degrees and alpha values for Ridge regression to find the best combination.

Practice

1.

What is the main purpose of using polynomial regression instead of simple linear regression?

easy

A. To fit curved relationships between variables

B. To reduce the number of features

C. To speed up training time

D. To handle missing data automatically

Solution

1. Step 1: Understand linear regression limitation

   Linear regression fits straight lines, which cannot capture curves in data.

2. Step 2: Role of polynomial regression

   Polynomial regression fits curved lines by adding powers of features, capturing non-linear patterns.

3. Final Answer:

   To fit curved relationships between variables -> Option A

4. Quick Check:

   Polynomial regression = curved fit [OK]

Hint: Polynomial regression fits curves, not just straight lines [OK]

Common Mistakes:

• Thinking polynomial regression reduces features
• Assuming it speeds up training
• Believing it handles missing data automatically

2.

Which of the following is the correct way to create a polynomial regression pipeline in Python using sklearn?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

pipeline = Pipeline([
    ('poly', PolynomialFeatures(degree=2)),
    ('linear', LinearRegression())
])

easy

A. pipeline = Pipeline([('poly', PolynomialFeatures(degree=2)), ('linear', LinearRegression())])

B. pipeline = Pipeline([('linear', LinearRegression()), ('poly', PolynomialFeatures(degree=2))])

C. pipeline = Pipeline([('poly', LinearRegression()), ('linear', PolynomialFeatures(degree=2))])

D. pipeline = Pipeline([('poly', PolynomialFeatures()), ('linear', LinearRegression(degree=2))])

Solution

1. Step 1: Order of pipeline steps

   PolynomialFeatures must come before LinearRegression to transform data first.

2. Step 2: Correct usage of classes and parameters

   PolynomialFeatures takes degree parameter; LinearRegression does not take degree.

3. Final Answer:

   pipeline = Pipeline([('poly', PolynomialFeatures(degree=2)), ('linear', LinearRegression())]) -> Option A

4. Quick Check:

   PolynomialFeatures before LinearRegression [OK]

Hint: Put PolynomialFeatures before LinearRegression in pipeline [OK]

Common Mistakes:

• Swapping order of pipeline steps
• Passing degree to LinearRegression
• Omitting degree in PolynomialFeatures

3.

Given the following code, what will print(y_pred) output?

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

X = np.array([[1], [2], [3]])
y = np.array([1, 4, 9])

pipeline = Pipeline([
    ('poly', PolynomialFeatures(degree=2)),
    ('linear', LinearRegression())
])
pipeline.fit(X, y)
y_pred = pipeline.predict(np.array([[4]]))
print(np.round(y_pred, 2))

medium

A. [10.0]

B. [8.0]

C. [4.0]

D. [16.0]

Solution

1. Step 1: Understand data and model

   X = [[1],[2],[3]] with y = [1,4,9] fits y = x^2 perfectly.

2. Step 2: Predict for X=4 using polynomial degree 2

   Model learns y = x^2, so prediction at 4 is 4^2 = 16.

3. Final Answer:

   [16.0] -> Option D

4. Quick Check:

   4 squared = 16 [OK]

Hint: Polynomial degree 2 fits squares; predict 4^2 = 16 [OK]

Common Mistakes:

• Ignoring polynomial transformation
• Predicting linear value instead of squared
• Rounding errors without np.round

4.

Identify the error in this polynomial regression pipeline code:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

pipeline = Pipeline([
    ('linear', LinearRegression()),
    ('poly', PolynomialFeatures(degree=3))
])

pipeline.fit(X_train, y_train)

medium

A. LinearRegression should not be used in pipeline

B. The order of pipeline steps is incorrect

C. PolynomialFeatures degree must be 2, not 3

D. Missing import for X_train and y_train

Solution

1. Step 1: Check pipeline step order

   PolynomialFeatures must come before LinearRegression to transform data first.

2. Step 2: Confirm degree and imports

   Degree 3 is valid; imports for data are assumed outside snippet.

3. Final Answer:

   The order of pipeline steps is incorrect -> Option B

4. Quick Check:

   PolynomialFeatures before LinearRegression [OK]

Hint: PolynomialFeatures must be first in pipeline [OK]

Common Mistakes:

• Swapping order of steps
• Thinking degree must be 2
• Confusing missing data imports with pipeline error

5.

You want to model a dataset with a complex curve. You try polynomial regression with degree=2 but the fit is poor. What is the best next step?

hard

A. Remove polynomial features and use linear regression only

B. Decrease the polynomial degree to avoid overfitting

C. Increase the polynomial degree to capture more complexity

D. Use degree=2 but reduce training data size

Solution

1. Step 1: Understand model complexity and fit

   Degree 2 polynomial may be too simple for complex curves, causing poor fit.

2. Step 2: Adjust polynomial degree

   Increasing degree allows model to fit more complex patterns, improving fit quality.

3. Final Answer:

   Increase the polynomial degree to capture more complexity -> Option C

4. Quick Check:

   Higher degree = better complex fit [OK]

Hint: Raise degree to fit complex curves better [OK]

Common Mistakes:

• Lowering degree when fit is poor
• Removing polynomial features unnecessarily
• Reducing data size instead of model complexity