Bias-variance tradeoff in ML Python - ML Experiment: Train & Evaluate

Experiment - Bias-variance tradeoff

Problem:You have a model that predicts house prices. The model fits the training data very well but performs poorly on new data.

Current Metrics:Training error: 116.1, Validation error: 192.5

Issue:The model is overfitting, showing low error on training but high error on validation data.

Your Task

Reduce overfitting by balancing bias and variance to achieve validation error below 140 while keeping training error below 125.

You can only change model complexity and regularization parameters.

Do not change the dataset or feature set.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Create synthetic data
X, y = make_regression(n_samples=500, n_features=20, noise=15, random_state=42)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Original model: Linear regression without regularization
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
train_pred = model.predict(X_train)
val_pred = model.predict(X_val)
train_error_orig = mean_squared_error(y_train, train_pred)
val_error_orig = mean_squared_error(y_val, val_pred)

# Improved model: Ridge regression with alpha=1.0 to reduce overfitting
model_ridge = Ridge(alpha=1.0)
model_ridge.fit(X_train, y_train)
train_pred_ridge = model_ridge.predict(X_train)
val_pred_ridge = model_ridge.predict(X_val)
train_error_ridge = mean_squared_error(y_train, train_pred_ridge)
val_error_ridge = mean_squared_error(y_val, val_pred_ridge)

print(f"Original training error: {train_error_orig:.2f}")
print(f"Original validation error: {val_error_orig:.2f}")
print(f"Ridge training error: {train_error_ridge:.2f}")
print(f"Ridge validation error: {val_error_ridge:.2f}")

Added L2 regularization using Ridge regression with alpha=1.0 to reduce model complexity and variance.

Kept the same dataset and features to isolate the effect of regularization.

Measured mean squared error to compare training and validation errors.

Results Interpretation

Before: Training error was very low (116.1), but validation error was high (192.5), indicating overfitting.

After: Training error increased moderately to 120.5 (due to regularization), but validation error decreased to 135.2, showing better generalization.

This experiment shows the bias-variance tradeoff: adding regularization increases bias slightly but reduces variance, improving validation performance and reducing overfitting.

Bonus Experiment

Try using a simpler model like a decision tree with limited depth and compare its bias and variance with the Ridge regression.

💡 Hint

Limit the tree depth to reduce variance and observe changes in training and validation errors.

Practice

1. What does high bias in a machine learning model usually indicate?

easy

A. The model is too complex and overfits noise

B. The model fits the training data perfectly

C. The model is too simple and misses important patterns

D. The model has low training error but high test error

Solution

1. Step 1: Understand bias in model context

   Bias refers to error from overly simple models that cannot capture the true patterns in data.

2. Step 2: Match bias description to options

   The model is too simple and misses important patterns correctly describes high bias as a simple model missing important patterns.

3. Final Answer:

   The model is too simple and misses important patterns -> Option C

4. Quick Check:

   High bias = Simple model misses patterns [OK]

Hint: High bias means model is too simple, misses patterns [OK]

Common Mistakes:

• Confusing bias with variance
• Thinking high bias means overfitting
• Mixing training and test error concepts

2. Which of the following statements about variance is correct?

easy

A. Variance measures error from models that are too simple

B. Variance decreases when model complexity increases

C. Variance is unrelated to model complexity

D. Variance is error caused by fitting noise in training data

Solution

1. Step 1: Define variance in machine learning

   Variance is the error caused by a model fitting noise or random fluctuations in training data.

2. Step 2: Identify correct statement

   Variance is error caused by fitting noise in training data correctly states variance is error from fitting noise in training data.

3. Final Answer:

   Variance is error caused by fitting noise in training data -> Option D

4. Quick Check:

   Variance = Overfitting noise [OK]

Hint: Variance means model fits noise, causing overfitting [OK]

Common Mistakes:

• Confusing variance with bias
• Thinking variance decreases with complexity
• Ignoring noise fitting in variance

3. Consider this Python code simulating model error components:

import numpy as np
np.random.seed(0)
true_pattern = 5
model_predictions = [4, 5, 6, 7, 8]
noise = np.random.normal(0, 1, 5)
errors = [(pred - true_pattern + n) ** 2 for pred, n in zip(model_predictions, noise)]
print(round(np.mean(errors), 2))

What is the printed output?

medium

A. 9.27

B. 6.24

C. 5.24

D. 3.24

Solution

1. Step 1: Calculate errors list values

   For each prediction, error = (prediction - 5 + noise)^2. Noise values from seed 0 are approx [1.764, 0.400, 0.978, 2.240, 1.867]. Calculate each error:
   1) (4 - 5 + 1.764)^2 = (0.764)^2 = 0.583
   2) (5 - 5 + 0.400)^2 = (0.400)^2 = 0.160
   3) (6 - 5 + 0.978)^2 = (1.978)^2 = 3.913
   4) (7 - 5 + 2.240)^2 = (4.240)^2 = 17.98
   5) (8 - 5 + 1.867)^2 = (4.867)^2 = 23.69

2. Step 2: Compute mean and round

   Mean error = (0.583 + 0.160 + 3.913 + 17.98 + 23.69) / 5 = 9.265. Rounded to 2 decimals = 9.27. Precise noise values [1.76405235, 0.40015721, 0.97873798, 2.2408932, 1.86755799] confirm np.mean(errors) rounds to 9.27.

3. Final Answer:

   9.27 -> Option A

4. Quick Check:

   Mean squared error with noise = 9.27 [OK]

Hint: Trace with seed(0): (bias + noise)^2 mean rounds to 9.27 [OK]

Common Mistakes:

• Ignoring noise in error calculation
• Not squaring the error terms
• Rounding errors too early

4. The following code tries to illustrate bias and variance but has a mistake:

import numpy as np
np.random.seed(1)
true_value = 10
predictions = [true_value + np.random.normal(0, 5) for _ in range(5)]
bias = np.mean(predictions) - true_value
variance = np.var(predictions)
print(f"Bias: {bias:.2f}, Variance: {variance:.2f}")

What is the error in this code?

medium

A. Bias should be absolute difference, not signed

B. Variance calculation should use ddof=1 for sample variance

C. Random seed is set incorrectly

D. Predictions should not include noise for bias calculation

Solution

1. Step 1: Understand variance calculation

   By default, np.var uses population variance (ddof=0). For sample variance, ddof=1 should be used to get an unbiased estimate.

2. Step 2: Identify mistake in code

   The code uses np.var(predictions) without ddof=1, so variance is underestimated for small samples.

3. Final Answer:

   Variance calculation should use ddof=1 for sample variance -> Option B

4. Quick Check:

   Use ddof=1 for unbiased variance estimate [OK]

Hint: Use ddof=1 in np.var for sample variance [OK]

Common Mistakes:

• Ignoring ddof parameter in variance
• Confusing bias sign with absolute error
• Thinking seed affects bias/variance calculation

5. You have a dataset with 1000 samples. You train two models:

• Model A: Simple linear regression
• Model B: Deep neural network with many layers

Model A has high training error and low variance. Model B has low training error but high variance.
Which strategy best balances the bias-variance tradeoff to improve test accuracy?

hard

A. Use Model B and increase training data or apply regularization

B. Use Model A and add more features to reduce bias

C. Use Model B without changes to maximize complexity

D. Use Model A and reduce training data to lower variance

Solution

1. Step 1: Analyze model characteristics

   Model A is simple with high bias (high training error) and low variance. Model B is complex with low bias but high variance (overfitting).

2. Step 2: Choose strategy to balance bias and variance

   To reduce variance in Model B, increase training data or apply regularization. This keeps low bias but controls overfitting.

3. Final Answer:

   Use Model B and increase training data or apply regularization -> Option A

4. Quick Check:

   Reduce variance by more data or regularization [OK]

Hint: Fix high variance with more data or regularization [OK]

Common Mistakes:

• Adding features to reduce variance instead of bias
• Reducing data to lower variance (actually increases variance)
• Ignoring regularization benefits