Bias-variance tradeoff in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Bias and Variance

Which statement best describes the relationship between bias and variance in a machine learning model?

A Low bias always leads to low variance in a model.

B High bias means the model fits the training data perfectly, while high variance means the model cannot fit the training data well.

C Bias and variance both measure how well the model performs on the training data only.

D High bias means the model is too simple and underfits, while high variance means the model is too complex and overfits.

💡 Hint

Think about how a model behaves when it is too simple or too complex.

Bias measures error from wrong assumptions (underfitting), variance measures error from sensitivity to training data (overfitting). High bias means underfitting, high variance means overfitting.

❓ Predict Output

intermediate

Effect of Model Complexity on Error

Consider the following Python code that simulates training and test errors for models of increasing complexity. What is the output of the print statement?

ML Python

import numpy as np
complexity = np.array([1, 2, 3, 4, 5])
train_error = 1 / complexity
test_error = 1 / complexity + 0.1 * complexity
print(f"Train error: {train_error}")
print(f"Test error: {test_error}")

A Train error decreases as complexity increases; test error decreases then increases.

B Both train and test errors increase as complexity increases.

C Train error increases as complexity increases; test error decreases as complexity increases.

D Train error stays constant; test error decreases as complexity increases.

💡 Hint

Look at how 1/complexity changes and how 0.1 * complexity affects test error.

Train error is 1/complexity, so it decreases as complexity grows. Test error adds a term increasing with complexity, so it first decreases then increases, showing bias-variance tradeoff.

❓ Hyperparameter

advanced

Choosing Regularization Strength

You train a linear regression model with L2 regularization (Ridge). Increasing the regularization parameter lambda will most likely:

A Decrease bias and increase variance, leading to more complex models.

B Increase bias and decrease variance, leading to simpler models.

C Increase both bias and variance, making the model unstable.

D Have no effect on bias or variance.

💡 Hint

Think about how regularization controls model complexity.

Increasing lambda penalizes large weights, simplifying the model, which increases bias but reduces variance.

❓ Metrics

advanced

Interpreting Learning Curves

You observe the following behavior in learning curves: training error is high and stable, but test error is high and does not improve with more data. What does this indicate about the model?

A The model has data leakage causing artificially low training error.

B The model has high variance and is overfitting the training data.

C The model has high bias and is underfitting the data.

D The model is perfectly balanced between bias and variance.

💡 Hint

Think about what high training error and high test error means.

High training error and high test error that does not improve with more data suggests the model is too simple and underfits, showing high bias.

🔧 Debug

expert

Diagnosing Model Behavior from Code

Given the following code snippet training a decision tree, what is the most likely cause of the model having high training error and high test error?

from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=1)
model.fit(X_train, y_train)
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"Train accuracy: {train_acc}")
print(f"Test accuracy: {test_acc}")

A The max_depth=1 is too low, causing high bias and underfitting.

B The max_depth=1 is too high, causing high variance and overfitting.

C The model is overfitting because max_depth is not set, so it grows fully.

D The training data is too small, causing unstable accuracy scores.

💡 Hint

Consider how tree depth affects model complexity and errors.

A decision tree with max_depth=1 is very simple, likely underfitting, which causes low training accuracy and low test accuracy due to high bias.

Practice

1. What does high bias in a machine learning model usually indicate?

easy

A. The model is too complex and overfits noise

B. The model fits the training data perfectly

C. The model is too simple and misses important patterns

D. The model has low training error but high test error

Solution

1. Step 1: Understand bias in model context

   Bias refers to error from overly simple models that cannot capture the true patterns in data.

2. Step 2: Match bias description to options

   The model is too simple and misses important patterns correctly describes high bias as a simple model missing important patterns.

3. Final Answer:

   The model is too simple and misses important patterns -> Option C

4. Quick Check:

   High bias = Simple model misses patterns [OK]

Hint: High bias means model is too simple, misses patterns [OK]

Common Mistakes:

• Confusing bias with variance
• Thinking high bias means overfitting
• Mixing training and test error concepts

2. Which of the following statements about variance is correct?

easy

A. Variance measures error from models that are too simple

B. Variance decreases when model complexity increases

C. Variance is unrelated to model complexity

D. Variance is error caused by fitting noise in training data

Solution

1. Step 1: Define variance in machine learning

   Variance is the error caused by a model fitting noise or random fluctuations in training data.

2. Step 2: Identify correct statement

   Variance is error caused by fitting noise in training data correctly states variance is error from fitting noise in training data.

3. Final Answer:

   Variance is error caused by fitting noise in training data -> Option D

4. Quick Check:

   Variance = Overfitting noise [OK]

Hint: Variance means model fits noise, causing overfitting [OK]

Common Mistakes:

• Confusing variance with bias
• Thinking variance decreases with complexity
• Ignoring noise fitting in variance

3. Consider this Python code simulating model error components:

import numpy as np
np.random.seed(0)
true_pattern = 5
model_predictions = [4, 5, 6, 7, 8]
noise = np.random.normal(0, 1, 5)
errors = [(pred - true_pattern + n) ** 2 for pred, n in zip(model_predictions, noise)]
print(round(np.mean(errors), 2))

What is the printed output?

medium

A. 9.27

B. 6.24

C. 5.24

D. 3.24

Solution

1. Step 1: Calculate errors list values

   For each prediction, error = (prediction - 5 + noise)^2. Noise values from seed 0 are approx [1.764, 0.400, 0.978, 2.240, 1.867]. Calculate each error:
   1) (4 - 5 + 1.764)^2 = (0.764)^2 = 0.583
   2) (5 - 5 + 0.400)^2 = (0.400)^2 = 0.160
   3) (6 - 5 + 0.978)^2 = (1.978)^2 = 3.913
   4) (7 - 5 + 2.240)^2 = (4.240)^2 = 17.98
   5) (8 - 5 + 1.867)^2 = (4.867)^2 = 23.69

2. Step 2: Compute mean and round

   Mean error = (0.583 + 0.160 + 3.913 + 17.98 + 23.69) / 5 = 9.265. Rounded to 2 decimals = 9.27. Precise noise values [1.76405235, 0.40015721, 0.97873798, 2.2408932, 1.86755799] confirm np.mean(errors) rounds to 9.27.

3. Final Answer:

   9.27 -> Option A

4. Quick Check:

   Mean squared error with noise = 9.27 [OK]

Hint: Trace with seed(0): (bias + noise)^2 mean rounds to 9.27 [OK]

Common Mistakes:

• Ignoring noise in error calculation
• Not squaring the error terms
• Rounding errors too early

4. The following code tries to illustrate bias and variance but has a mistake:

import numpy as np
np.random.seed(1)
true_value = 10
predictions = [true_value + np.random.normal(0, 5) for _ in range(5)]
bias = np.mean(predictions) - true_value
variance = np.var(predictions)
print(f"Bias: {bias:.2f}, Variance: {variance:.2f}")

What is the error in this code?

medium

A. Bias should be absolute difference, not signed

B. Variance calculation should use ddof=1 for sample variance

C. Random seed is set incorrectly

D. Predictions should not include noise for bias calculation

Solution

1. Step 1: Understand variance calculation

   By default, np.var uses population variance (ddof=0). For sample variance, ddof=1 should be used to get an unbiased estimate.

2. Step 2: Identify mistake in code

   The code uses np.var(predictions) without ddof=1, so variance is underestimated for small samples.

3. Final Answer:

   Variance calculation should use ddof=1 for sample variance -> Option B

4. Quick Check:

   Use ddof=1 for unbiased variance estimate [OK]

Hint: Use ddof=1 in np.var for sample variance [OK]

Common Mistakes:

• Ignoring ddof parameter in variance
• Confusing bias sign with absolute error
• Thinking seed affects bias/variance calculation

5. You have a dataset with 1000 samples. You train two models:

• Model A: Simple linear regression
• Model B: Deep neural network with many layers

Model A has high training error and low variance. Model B has low training error but high variance.
Which strategy best balances the bias-variance tradeoff to improve test accuracy?

hard

A. Use Model B and increase training data or apply regularization

B. Use Model A and add more features to reduce bias

C. Use Model B without changes to maximize complexity

D. Use Model A and reduce training data to lower variance

Solution

1. Step 1: Analyze model characteristics

   Model A is simple with high bias (high training error) and low variance. Model B is complex with low bias but high variance (overfitting).

2. Step 2: Choose strategy to balance bias and variance

   To reduce variance in Model B, increase training data or apply regularization. This keeps low bias but controls overfitting.

3. Final Answer:

   Use Model B and increase training data or apply regularization -> Option A

4. Quick Check:

   Reduce variance by more data or regularization [OK]

Hint: Fix high variance with more data or regularization [OK]

Common Mistakes:

• Adding features to reduce variance instead of bias
• Reducing data to lower variance (actually increases variance)
• Ignoring regularization benefits