Why regression predicts continuous values in ML Python - Experiment to Prove It

Experiment - Why regression predicts continuous values

Problem:We want to predict house prices based on features like size and number of rooms. The current model predicts prices but sometimes gives values that are too close to training data and does not generalize well.

Current Metrics:Training loss: 0.02, Training RMSE: 500, Validation loss: 0.15, Validation RMSE: 1500

Issue:The model overfits the training data and does not predict continuous values smoothly for new data.

Your Task

Improve the regression model so it predicts continuous house prices more accurately and reduces overfitting, aiming for validation RMSE below 1000 while keeping training RMSE below 800.

Keep the model as a regression model (predict continuous values).

Use the same dataset and features.

Do not change the problem to classification.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error

# Simulated dataset: features (size, rooms), target (price)
np.random.seed(42)
X = np.random.rand(500, 2) * [2000, 5]  # size in sqft, rooms
true_coef = np.array([150, 10000])
y = X @ true_coef + np.random.randn(500) * 10000  # price with noise

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Use Ridge regression (L2 regularization) to reduce overfitting
model = Ridge(alpha=1.0)  # alpha controls regularization strength
model.fit(X_train, y_train)

# Predictions
train_preds = model.predict(X_train)
val_preds = model.predict(X_val)

# Metrics
train_rmse = mean_squared_error(y_train, train_preds, squared=False)
val_rmse = mean_squared_error(y_val, val_preds, squared=False)

print(f"Training RMSE: {train_rmse:.2f}")
print(f"Validation RMSE: {val_rmse:.2f}")

Replaced a complex model with Ridge regression to add L2 regularization.

Used mean squared error loss suitable for continuous value prediction.

Kept the problem as regression to predict continuous house prices.

Results Interpretation

Before: Training RMSE = 500, Validation RMSE = 1500 (overfitting)

After: Training RMSE = 720, Validation RMSE = 950 (better generalization)

Regression models predict continuous values by learning a function that fits the data points smoothly. Adding regularization helps the model avoid memorizing training data and improves prediction on new data.

Bonus Experiment

Try using polynomial features with Ridge regression to see if capturing non-linear relationships improves continuous value prediction.

💡 Hint

Use sklearn's PolynomialFeatures to transform input features before fitting the Ridge model.

Practice

1. Why does regression predict continuous values instead of categories?

easy

A. Because it predicts yes or no answers

B. Because it groups data into fixed classes

C. Because it only works with text data

D. Because it estimates quantities that can take any value within a range

Solution

1. Step 1: Understand regression's purpose

   Regression is designed to predict numbers that can vary continuously, like prices or temperatures.

2. Step 2: Compare with classification

   Classification predicts fixed groups or categories, not continuous values.

3. Final Answer:

   Because it estimates quantities that can take any value within a range -> Option D

4. Quick Check:

   Regression predicts continuous values = Because it estimates quantities [OK]

Hint: Regression predicts numbers, classification predicts groups [OK]

Common Mistakes:

• Confusing regression with classification
• Thinking regression predicts categories
• Assuming regression only works with discrete values

2. Which of the following is the correct way to define a regression target variable in Python?

easy

A. target = [3.5, 7.2, 1.8]

B. target = ['cat', 'dog', 'bird']

C. target = ['red', 'blue', 'green']

D. target = ['yes', 'no', 'maybe']

Solution

1. Step 1: Identify regression target type

   Regression targets are continuous numbers, like 3.5 or 7.2.

2. Step 2: Check options for continuous values

   Only target = [3.5, 7.2, 1.8] contains numbers suitable for regression.

3. Final Answer:

   target = [3.5, 7.2, 1.8] -> Option A

4. Quick Check:

   Regression targets are numbers = target with floats [OK]

Hint: Regression targets are numeric lists, not categories [OK]

Common Mistakes:

• Using categorical strings as regression targets
• Confusing classification labels with regression targets
• Using boolean or text values for regression

3. You want to predict house prices using regression. Your dataset has features like size (sq ft), number of rooms, and age of the house. Why is regression the right choice here?

easy

A. Because house prices belong to fixed categories like cheap or expensive

B. Because house prices are continuous values that can vary widely

C. Because regression only works with text data

D. Because regression predicts yes/no answers

Solution

1. Step 1: Identify target variable type

   House prices are numbers that can take many values, not fixed groups.

2. Step 2: Match problem to regression

   Regression predicts continuous numeric values, perfect for price prediction.

3. Final Answer:

   Because house prices are continuous values that can vary widely -> Option B

4. Quick Check:

   Price prediction = continuous value regression [OK]

Hint: Predict continuous numbers like prices with regression [OK]

Common Mistakes:

• Treating prices as categories
• Choosing classification instead of regression
• Confusing regression with yes/no prediction

4. Given this simple regression model in Python using scikit-learn:

from sklearn.linear_model import LinearRegression
X = [[1], [2], [3], [4]]
y = [2.0, 4.1, 6.0, 8.1]
model = LinearRegression()
model.fit(X, y)
pred = model.predict([[5]])
print(round(pred[0], 1))

What will be the printed output?

medium

A. 10.1

B. 8.5

C. 9.9

D. 10.0

Solution

1. Step 1: Understand the data and model

   The data roughly follows y = 2 * x with small noise. The model fits a line close to y=2x.

2. Step 2: Predict for x=5

   Prediction is about 2 * 5 = 10, but slight noise shifts it to around 10.1.

3. Final Answer:

   10.1 -> Option A

4. Quick Check:

   Linear fit predicts ~10.1 for input 5 [OK]

Hint: Regression predicts near trend line values, not exact multiples [OK]

Common Mistakes:

• Assuming exact multiples without noise
• Confusing classification output with regression output
• Rounding incorrectly

5. You wrote this regression code but get an error:

from sklearn.linear_model import LinearRegression
X = [1, 2, 3, 4]
y = [2.0, 4.1, 6.0, 8.1]
model = LinearRegression()
model.fit(X, y)

What is the error and how to fix it?

medium

A. model.fit needs no arguments, remove X and y

B. y should be strings, fix with y = ['2.0', '4.1', '6.0', '8.1']

C. X should be a 2D array, fix with X = [[1], [2], [3], [4]]

D. LinearRegression only works with categorical data

Solution

1. Step 1: Identify input shape error

   scikit-learn expects X as 2D array (list of lists), but X is 1D list.

2. Step 2: Fix input shape

   Wrap each value in a list: X = [[1], [2], [3], [4]] to make it 2D.

3. Final Answer:

   X should be a 2D array, fix with X = [[1], [2], [3], [4]] -> Option C

4. Quick Check:

   Input X must be 2D array for regression [OK]

Hint: Regression input X must be 2D array, not 1D list [OK]

Common Mistakes:

• Passing 1D list instead of 2D array for X
• Changing y to strings incorrectly
• Removing required fit arguments