Why advanced regression handles non-linearity in ML Python - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why can polynomial regression model non-linear data?

Polynomial regression can fit curves instead of straight lines. Why does this help with non-linear data?

A Because it adds powers of input features, allowing the model to capture curves.

B Because it removes noise from data, making it linear.

C Because it uses only linear combinations of features without transformation.

D Because it reduces the number of features to avoid overfitting.

💡 Hint

Think about how adding squared or cubic terms changes the shape of the prediction.

Polynomial regression adds powers of input features (like squared or cubic terms). This lets the model fit curved lines, which helps capture non-linear relationships in data.

❓ Model Choice

intermediate

Which regression model best handles complex non-linear patterns?

You have data with complex curves and interactions. Which regression model is best to capture these patterns?

A Linear regression

B Polynomial regression with degree 3

C Simple mean prediction

D Decision tree regression

💡 Hint

Think about models that split data into regions and fit different values in each.

Decision tree regression splits data into regions and fits simple models in each. This allows it to capture complex non-linear patterns better than polynomial regression of fixed degree.

❓ Predict Output

advanced

Output of polynomial regression prediction

What is the predicted value from this polynomial regression model?

ML Python

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([[2]])
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
model = LinearRegression()
model.coef_ = np.array([3, 4])
model.intercept_ = 5
prediction = model.predict(X_poly)
print(prediction[0])

A 19.0

B 17.0

C 27.0

D 23.0

💡 Hint

Calculate 3*x + 4*x^2 + 5 for x=2.

The polynomial features for x=2 are [2, 4]. The prediction is 5 + 3*2 + 4*4 = 5 + 6 + 16 = 27.0.

❓ Metrics

advanced

Which metric best shows improvement in non-linear regression?

You compare linear regression and polynomial regression on the same dataset. Which metric best shows that polynomial regression fits non-linear data better?

A Higher Mean Squared Error (MSE)

B Lower Mean Squared Error (MSE)

C Higher training loss

D Lower number of features

💡 Hint

Think about what a good fit means for error values.

A lower Mean Squared Error means the model predictions are closer to actual values, showing better fit. Polynomial regression usually lowers MSE on non-linear data compared to linear regression.

🔧 Debug

expert

Why does this kernel ridge regression code fail to capture non-linearity?

Given this code snippet, why does the kernel ridge regression model fail to capture non-linear patterns?

ML Python

from sklearn.kernel_ridge import KernelRidge
import numpy as np

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([1, 4, 9, 16, 25])  # y = x^2

model = KernelRidge(alpha=1.0, kernel='linear')
model.fit(X, y)
pred = model.predict(np.array([[6]]))
print(round(pred[0], 2))

A Because the kernel is linear, it cannot model the quadratic relationship.

B Because alpha is too high, causing overfitting.

C Because the input X is not scaled.

D Because the model is missing the intercept term.

💡 Hint

Think about what the kernel function does in kernel ridge regression.

The linear kernel treats the problem like linear regression, so it cannot capture the quadratic pattern y = x^2. Using a non-linear kernel like 'rbf' would help model non-linearity.

Practice

1. Why do advanced regression models handle non-linearity better than simple linear regression?

easy

A. Because they only use one feature at a time

B. Because they ignore data points that don't fit a line

C. Because they can model complex curved relationships in data

D. Because they always use fewer data points

Solution

1. Step 1: Understand simple linear regression limits

   Simple linear regression fits a straight line, so it cannot capture curves or bends in data.

2. Step 2: Recognize advanced regression capabilities

   Advanced regression models like decision trees or polynomial regression can fit curves and complex patterns.

3. Final Answer:

   Because they can model complex curved relationships in data -> Option C

4. Quick Check:

   Advanced regression models handle curves [OK]

Hint: Advanced regression fits curves, not just straight lines [OK]

Common Mistakes:

• Thinking advanced regression ignores data points
• Believing advanced regression uses fewer data points
• Assuming advanced regression only uses one feature

2. Which of the following is the correct way to create a polynomial regression model in Python using scikit-learn?

easy

A. from sklearn.preprocessing import PolynomialFeatures; poly = PolynomialFeatures(degree=2); X_poly = poly.fit_transform(X)

B. from sklearn.tree import DecisionTreeRegressor; model = DecisionTreeRegressor(); model.fit(X, y)

C. from sklearn.cluster import KMeans; model = KMeans(); model.fit(X)

D. from sklearn.linear_model import LinearRegression; model = LinearRegression(); model.fit(X_poly, y)

Solution

1. Step 1: Identify polynomial feature creation

   Polynomial regression requires transforming features using PolynomialFeatures to add powers of features.

2. Step 2: Recognize correct syntax for polynomial transformation

   from sklearn.preprocessing import PolynomialFeatures; poly = PolynomialFeatures(degree=2); X_poly = poly.fit_transform(X) correctly imports PolynomialFeatures and transforms X to X_poly for regression.

3. Final Answer:

   from sklearn.preprocessing import PolynomialFeatures; poly = PolynomialFeatures(degree=2); X_poly = poly.fit_transform(X) -> Option A

4. Quick Check:

   PolynomialFeatures creates polynomial features [OK]

Hint: Polynomial regression needs PolynomialFeatures to transform data [OK]

Common Mistakes:

• Confusing decision tree with polynomial regression
• Using clustering models for regression tasks
• Not transforming features before fitting polynomial regression

3. Given the code below, what will be the output of print(predictions)?

from sklearn.tree import DecisionTreeRegressor
X = [[1], [2], [3], [4], [5]]
y = [1, 4, 9, 16, 25]
model = DecisionTreeRegressor()
model.fit(X, y)
predictions = model.predict([[6]])
print(predictions)

medium

A. [16]

B. [36]

C. [9]

D. [25]

Solution

1. Step 1: Understand decision tree prediction behavior

   Decision trees predict by assigning the output of the closest training leaf node, not extrapolating beyond training data.

2. Step 2: Check training data and prediction input

   Input 6 is beyond training max 5, so prediction will be the leaf value for closest known input, which is 5 with output 25.

3. Final Answer:

   [25] -> Option D

4. Quick Check:

   Decision tree predicts closest leaf value = 25 [OK]

Hint: Decision trees do not extrapolate; predict closest known value [OK]

Common Mistakes:

• Assuming decision tree extrapolates like polynomial regression
• Expecting exact square of 6 (36) as output
• Confusing prediction with training labels

4. The following code tries to fit a polynomial regression but gives an error. What is the mistake?

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
X = [[1], [2], [3], [4]]
y = [1, 4, 9, 16]
model = LinearRegression()
X_poly = PolynomialFeatures(degree=2)
model.fit(X_poly, y)

medium

A. LinearRegression cannot fit polynomial data

B. X_poly is a class, not transformed data; need to call fit_transform on X

C. y should be a 2D array, not 1D

D. Degree should be 1 for polynomial features

Solution

1. Step 1: Identify how PolynomialFeatures is used

   PolynomialFeatures is a transformer class; it needs to be applied to X using fit_transform to create polynomial features.

2. Step 2: Spot the error in code

   Code assigns X_poly to the class instance, not the transformed data. The model.fit expects numeric array, not a class object.

3. Final Answer:

   X_poly is a class, not transformed data; need to call fit_transform on X -> Option B

4. Quick Check:

   Call fit_transform on X before fitting model [OK]

Hint: Call fit_transform on X before fitting model [OK]

Common Mistakes:

• Passing transformer class instead of transformed data
• Thinking LinearRegression can't fit polynomial features
• Misunderstanding y shape requirements

5. You have a dataset where the target variable changes in a complex curve with two features. Which approach best handles this non-linearity and why?

hard

A. Polynomial regression of degree 3 can model complex curves with multiple features

B. Simple linear regression will miss the curve

C. Decision tree with max depth 2 is too shallow to capture complexity

D. Dropping features reduces information and won't help non-linearity

Solution

1. Step 1: Analyze model capabilities for non-linearity

   Simple linear regression cannot model curves; decision tree with low depth may underfit; dropping features loses info.

2. Step 2: Evaluate polynomial regression for multiple features

   Polynomial regression with degree 3 creates interaction and power terms, capturing complex curves in multiple features.

3. Final Answer:

   Polynomial regression of degree 3 can model complex curves with multiple features -> Option A

4. Quick Check:

   Degree 3 polynomial regression models complex curves [OK]

Hint: Higher degree polynomial regression models complex curves well [OK]

Common Mistakes:

• Choosing shallow decision trees that underfit
• Dropping features reduces model power
• Using simple linear regression for curved data