Multiple linear regression in ML Python - Deep Dive

Overview - Multiple linear regression

What is it?

Multiple linear regression is a way to predict a number using several input factors. It finds a straight line that best fits the data points in many dimensions. Each input factor has a weight that shows how much it affects the prediction. This helps us understand and predict outcomes based on multiple causes.

Why it matters

Without multiple linear regression, we would struggle to understand how several things together influence a result. For example, predicting house prices depends on size, location, and age, not just one factor. This method helps businesses, scientists, and governments make better decisions by seeing the combined effect of many variables.

Where it fits

Before learning multiple linear regression, you should understand simple linear regression and basic algebra. After this, you can explore more complex models like polynomial regression, regularization techniques, and machine learning algorithms such as decision trees and neural networks.

Mental Model

Core Idea

Multiple linear regression finds the best flat surface that fits data points by weighing each input factor to predict an outcome.

Think of it like...

Imagine you are mixing paint colors to get a specific shade. Each color (input factor) contributes a certain amount (weight) to the final color (prediction). Multiple linear regression figures out how much of each color to mix to match the target shade.

Data points in 3D space
  ●  ●
    ●    ●
      ●      ●
       ┌─────────────┐
       │  Plane fit  │
       └─────────────┘
Inputs: x1, x2 → Output: y
Weights: w1, w2
Prediction: y = w1*x1 + w2*x2 + bias

Build-Up - 7 Steps

1

Foundation

Understanding simple linear regression

Concept: Learn how one input predicts one output using a straight line.

Simple linear regression fits a line y = wx + b to data points with one input x and one output y. It finds the best w (weight) and b (bias) to minimize the difference between predicted and actual y values.

Result

A line that best fits the data points, allowing prediction of y from x.

Understanding simple linear regression builds the base for handling multiple inputs by extending the idea of fitting a line to fitting a flat surface.

2

Foundation

Concept of multiple inputs and weights

Concept: Extend from one input to many inputs, each with its own weight.

Instead of one input x, we have multiple inputs x1, x2, ..., xn. Each input has a weight w1, w2, ..., wn. The prediction is the sum of each input times its weight plus a bias: y = w1*x1 + w2*x2 + ... + wn*xn + b.

Result

A formula that combines many inputs to predict one output.

Knowing that each input has its own weight helps us see how different factors contribute differently to the prediction.

3

Intermediate

Fitting the model with least squares

🤔Before reading on: Do you think the best fit minimizes the sum of absolute errors or the sum of squared errors? Commit to your answer.

Concept: Learn how to find the best weights by minimizing the squared differences between predictions and actual values.

The least squares method finds weights that minimize the sum of squared errors: sum((y_actual - y_predicted)^2). Squaring errors penalizes large mistakes more and makes the math easier to solve.

Result

Weights that produce the smallest total squared error on the training data.

Understanding least squares explains why the model fits the data in a way that balances all errors, not just some.

4

Intermediate

Interpreting coefficients and bias

🤔Before reading on: Does a positive weight always mean the input increases the output? Commit to yes or no.

Concept: Learn what the weights and bias mean in real terms.

Each weight shows how much the output changes when that input increases by one unit, holding others constant. The bias is the predicted output when all inputs are zero. Positive weights increase output; negative weights decrease it.

Result

Ability to explain how each input affects the prediction.

Knowing how to interpret coefficients helps turn the model from a black box into a tool for understanding relationships.

5

Intermediate

Checking model quality with R-squared

🤔Before reading on: Is a higher R-squared always better? Commit to yes or no.

Concept: Learn a metric that shows how well the model explains the data.

R-squared measures the proportion of variance in the output explained by the inputs. It ranges from 0 to 1. Closer to 1 means better fit. It helps judge if the model is useful.

Result

A number that summarizes model accuracy and usefulness.

Understanding R-squared helps decide if the model is good enough or needs improvement.

6

Advanced

Dealing with multicollinearity

🤔Before reading on: Do you think highly correlated inputs make the model more stable or less stable? Commit to your answer.

Concept: Learn why having inputs that are very similar causes problems in the model.

Multicollinearity happens when inputs are strongly correlated. It makes weights unstable and hard to interpret because the model can't tell which input is responsible for changes. This can lead to unreliable predictions.

Result

Awareness of when the model's weights may be misleading or unstable.

Knowing about multicollinearity prevents trusting coefficients blindly and guides better input selection.

7

Expert

Regularization to improve model robustness

🤔Before reading on: Does adding regularization increase or decrease model complexity? Commit to your answer.

Concept: Learn how to add penalties to weights to avoid overfitting and improve prediction on new data.

Regularization adds a penalty term to the loss function, like L1 (lasso) or L2 (ridge), which shrinks weights towards zero. This reduces overfitting, especially when many inputs exist or data is noisy.

Result

Models that generalize better and have simpler, more stable weights.

Understanding regularization reveals how to balance fitting data well and keeping the model simple for real-world use.

Under the Hood

Multiple linear regression solves a system of equations derived from minimizing the sum of squared errors. It uses matrix algebra to find weights that best fit the data. Internally, it computes (X^T X)^-1 X^T y, where X is the input matrix and y the output vector. This finds the exact weights for the best linear fit if X^T X is invertible.

Why designed this way?

The least squares approach was chosen because it provides a unique, mathematically tractable solution with good statistical properties. Alternatives like minimizing absolute errors are harder to solve analytically. Matrix algebra allows efficient computation even with many inputs.

Inputs X (matrix) ──▶ Multiply by weights w (vector) ──▶ Sum plus bias b ──▶ Prediction y_hat
          │
          ▼
   Compare with actual y
          │
          ▼
   Calculate squared errors
          │
          ▼
   Minimize sum of squared errors
          │
          ▼
   Solve for weights w using matrix inverse

Myth Busters - 4 Common Misconceptions

Quick: Does a high R-squared guarantee the model predicts well on new data? Commit to yes or no.

Common Belief: A high R-squared means the model is perfect and will predict new data accurately.

Reality: High R-squared only shows good fit on training data; the model can still perform poorly on new data due to overfitting.

Why it matters: Relying solely on R-squared can lead to trusting models that fail in real-world predictions, causing wrong decisions.

Quick: If one input has zero weight, does it mean that input has no relationship with the output? Commit to yes or no.

Common Belief: If a weight is zero, that input does not affect the output at all.

Reality: A zero weight can result from multicollinearity or model constraints, not necessarily no relationship. The input might be important but redundant with others.

Why it matters: Misinterpreting zero weights can cause ignoring important factors and misunderstanding the system.

Quick: Does multiple linear regression work well with any number of inputs without problems? Commit to yes or no.

Common Belief: You can add as many inputs as you want, and the model will always improve.

Reality: Adding too many inputs, especially irrelevant ones, can cause overfitting and unstable weights, reducing model usefulness.

Why it matters: Blindly adding inputs wastes resources and harms prediction quality, leading to poor decisions.

Quick: Is the relationship between inputs and output always linear in multiple linear regression? Commit to yes or no.

Common Belief: Multiple linear regression can model any kind of relationship between inputs and output.

Reality: It only models linear relationships; nonlinear patterns require other methods or feature transformations.

Why it matters: Using linear regression on nonlinear data leads to poor predictions and misunderstanding of the problem.

Expert Zone

1

Regularization not only prevents overfitting but also helps with multicollinearity by shrinking correlated weights.

2

The bias term can be absorbed into the weights by adding a constant input of 1, simplifying matrix calculations.

3

Interpretation of coefficients depends on input scaling; standardizing inputs before fitting improves stability and comparability.

When NOT to use

Avoid multiple linear regression when relationships are strongly nonlinear or when inputs interact in complex ways. Use polynomial regression, decision trees, or neural networks instead. Also, if inputs are categorical without proper encoding, linear regression is not suitable.

Production Patterns

In real systems, multiple linear regression is often combined with feature selection and regularization. It is used for quick, interpretable models in finance, healthcare, and marketing. Pipelines automate data cleaning, scaling, and model fitting for reliable deployment.

Connections

Principal Component Analysis (PCA)

Builds-on and complements

PCA reduces input dimensions before regression, helping avoid multicollinearity and improving model stability.

Econometrics

Same pattern applied in economics

Multiple linear regression is foundational in econometrics for modeling economic relationships and policy effects.

Linear Equations in Algebra

Mathematical foundation

Understanding solving systems of linear equations helps grasp how regression finds weights mathematically.

Common Pitfalls

#1 Ignoring input scaling causes unstable weights and poor interpretation.

Wrong approach:

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X, y) # X has features with very different scales

Correct approach:

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
model = LinearRegression()
model.fit(X_scaled, y)

Root cause:

Different input scales cause some weights to dominate and numerical instability in calculations.

#2 Using multiple linear regression on nonlinear data without transformation.

Wrong approach:

model.fit(X, y) # X and y have nonlinear relationship, no feature engineering

Correct approach:

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
model.fit(X_poly, y)

Root cause:

Linear regression assumes linearity; nonlinear patterns require feature transformations.

#3 Including highly correlated inputs without checking multicollinearity.

Wrong approach:

X = pd.DataFrame({'x1': data1, 'x2': data1 * 0.99})
model.fit(X, y)

Correct approach:

from statsmodels.stats.outliers_influence import variance_inflation_factor
vif = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
# Remove inputs with high VIF before fitting
model.fit(X_filtered, y)

Root cause:

Ignoring multicollinearity leads to unstable and misleading coefficient estimates.

Key Takeaways

Multiple linear regression predicts an outcome using several inputs by fitting a weighted sum plus bias.

It uses least squares to find weights that minimize the total squared prediction error.

Interpreting weights reveals how each input influences the output, but beware of multicollinearity.

Regularization improves model stability and prevents overfitting, especially with many inputs.

Understanding its limits and assumptions helps choose the right model for your data.