Regression analysis in MATLAB - Deep Dive

Overview - Regression analysis

What is it?

Regression analysis is a way to find the relationship between one thing you want to predict and one or more things you know. It helps you understand how changes in the known things affect the thing you want to predict. For example, it can show how house size and age affect its price. It uses math to draw a line or curve that best fits the data points.

Why it matters

Without regression analysis, we would struggle to make predictions or understand relationships in data. It helps businesses, scientists, and governments make decisions based on data trends. For example, predicting sales, understanding health risks, or forecasting weather would be much harder without it. It turns raw data into useful insights.

Where it fits

Before learning regression, you should know basic statistics like mean and variance, and understand simple plotting. After regression, you can learn more complex models like classification, time series forecasting, or machine learning techniques.

Mental Model

Core Idea

Regression analysis finds the best mathematical line or curve that explains how one or more inputs relate to an output.

Think of it like...

Imagine trying to draw a straight line through a scatter of points on a paper so that the line is as close as possible to all points. This line helps you guess where new points might be.

Data points (x,y):
  *   *    *
 *  *  *  *  *
  *    *  *

Best fit line:
  ──────────────
  (line through the middle of points)

Build-Up - 7 Steps

1

Foundation

Understanding variables and data points

Concept: Learn what variables are and how data points represent observations.

In regression, we have input variables (called predictors) and an output variable (called response). Each observation is a data point with values for these variables. For example, a house's size (input) and its price (output).

Result

You can identify what you want to predict and what information you have to predict it.

Knowing the difference between inputs and outputs is the base for building any predictive model.

2

Foundation

Plotting data to see relationships

Concept: Visualizing data helps understand if a relationship exists between variables.

Plot the input variable on the x-axis and the output on the y-axis. Look for patterns like a line or curve. For example, in MATLAB: x = [1 2 3 4]; y = [2 4 6 8]; plot(x,y,'o');

Result

You see if the points roughly form a line or curve, suggesting a relationship.

Visual patterns guide you on what kind of regression model to use.

3

Intermediate

Fitting a simple linear regression line

🤔Before reading on: do you think the best line always passes through all points exactly? Commit to your answer.

Concept: Learn how to find the line that best fits data by minimizing errors.

Simple linear regression fits a line y = mx + b that minimizes the sum of squared differences between actual and predicted y values. In MATLAB, use: mdl = fitlm(x,y); plot(mdl);

Result

You get a line that best represents the trend in data, even if it doesn't pass through all points.

Understanding that the best fit balances all errors helps grasp why exact fits are rare and often not desired.

4

Intermediate

Interpreting regression coefficients

🤔Before reading on: does a positive slope always mean a strong relationship? Commit to your answer.

Concept: Coefficients tell how much the output changes when an input changes.

The slope (m) shows how much y changes for a one-unit change in x. The intercept (b) is the predicted y when x is zero. In MATLAB, access coefficients by mdl.Coefficients.Estimate.

Result

You can explain how inputs affect outputs numerically.

Knowing coefficients lets you quantify relationships, not just see them visually.

5

Intermediate

Extending to multiple regression variables

🤔Before reading on: do you think adding more variables always improves predictions? Commit to your answer.

Concept: Regression can use many inputs to predict one output.

Multiple regression fits a model like y = b0 + b1x1 + b2x2 + ... In MATLAB, use: mdl = fitlm(table_of_data); where table_of_data has multiple columns.

Result

You get a model that considers several factors at once.

Understanding multiple inputs reflects real-world complexity where many factors influence outcomes.

6

Advanced

Checking model quality with R-squared and residuals

🤔Before reading on: does a higher R-squared always mean a better model? Commit to your answer.

Concept: Learn how to measure how well the model fits data and spot problems.

R-squared shows the proportion of variance explained by the model. Residuals are differences between actual and predicted values. In MATLAB, use mdl.Rsquared and plotResiduals(mdl).

Result

You can judge if the model explains data well and if errors are random.

Knowing model quality metrics prevents trusting poor models and guides improvements.

7

Expert

Understanding assumptions and their impact

🤔Before reading on: do you think regression works well even if data errors are not random? Commit to your answer.

Concept: Regression relies on assumptions like linearity, independence, and normality of errors.

If assumptions fail, results can be misleading. For example, correlated errors or non-linear relationships require different methods. MATLAB tools like plotDiagnostics(mdl) help check assumptions.

Result

You learn when regression results are trustworthy and when to use other methods.

Understanding assumptions is key to applying regression correctly and avoiding wrong conclusions.

Under the Hood

Regression analysis works by finding parameters (like slope and intercept) that minimize the sum of squared differences between observed outputs and predicted outputs. This is done using mathematical optimization techniques like least squares. Internally, MATLAB solves equations derived from calculus to find these parameters efficiently.

Why designed this way?

Least squares regression was designed to provide a simple, mathematically tractable way to find the best fit line. It balances errors across all points and has nice statistical properties. Alternatives like minimizing absolute errors exist but are harder to compute and less stable.

Data points
  ┌─────────────┐
  │ *   *   *   │
  │   *     *   │
  │ *     *     │
  └─────────────┘

Regression line
  ──────────────

Error calculation
  For each point:
    vertical distance to line
  Sum of squared distances minimized

Myth Busters - 4 Common Misconceptions

Quick: does a perfect correlation mean one variable causes the other? Commit yes or no.

Common Belief: If two variables have a strong regression relationship, one causes the other.

Reality: Correlation does not imply causation; other factors or coincidence can cause relationships.

Why it matters: Mistaking correlation for causation can lead to wrong decisions, like blaming the wrong cause for a problem.

Quick: does adding more variables always improve regression predictions? Commit yes or no.

Common Belief: More variables always make the regression model better.

Reality: Adding irrelevant variables can cause overfitting, making the model worse on new data.

Why it matters: Overfitting leads to poor predictions and wasted effort on unnecessary data.

Quick: does a high R-squared guarantee a good model? Commit yes or no.

Common Belief: A high R-squared means the model is perfect and reliable.

Reality: High R-squared can occur even with wrong models or overfitting; it doesn't guarantee validity.

Why it matters: Relying only on R-squared can hide model problems and cause false confidence.

Quick: can regression handle any kind of data relationship? Commit yes or no.

Common Belief: Regression works well for all data relationships, linear or not.

Reality: Regression assumes linearity; non-linear relationships need special models or transformations.

Why it matters: Using linear regression on non-linear data leads to poor fits and wrong conclusions.

Expert Zone

1

Regularization techniques like Ridge or Lasso add penalties to coefficients to prevent overfitting, which is not obvious in basic regression.

2

Multicollinearity, where input variables are highly correlated, can distort coefficient estimates and is often overlooked.

3

Heteroscedasticity, when error variance changes with input values, violates assumptions and affects inference but is subtle to detect.

When NOT to use

Avoid linear regression when data relationships are clearly non-linear or when errors are not independent and identically distributed. Use alternatives like generalized additive models, decision trees, or neural networks instead.

Production Patterns

In real-world systems, regression models are often combined with feature engineering, cross-validation, and automated hyperparameter tuning. They are used for forecasting sales, risk assessment, and quality control, often integrated into dashboards for decision makers.

Connections

Machine Learning Classification

Builds-on

Understanding regression helps grasp classification models since both predict outcomes from inputs, but classification predicts categories instead of numbers.

Econometrics

Same pattern

Regression is the core tool in econometrics to analyze economic data and test theories, showing how data science and economics deeply connect.

Physics - Linear Systems

Same pattern

Regression’s least squares method is mathematically similar to solving linear systems in physics, revealing a shared foundation across fields.

Common Pitfalls

#1 Ignoring the assumptions of regression leads to wrong conclusions.

Wrong approach:

mdl = fitlm(x,y);
% No checks for residual patterns or error distribution

Correct approach:

mdl = fitlm(x,y);
plotResiduals(mdl);
% Check residuals to validate assumptions

Root cause:

Learners often treat regression as a black box without validating if the model fits the data properly.

#2 Using regression to predict outside the range of data (extrapolation) without caution.

Wrong approach:

predict(mdl, [100]); % Predicting for x=100 when training data was 1-10

Correct approach:

% Avoid predictions far outside training data range or use models designed for extrapolation

Root cause:

Misunderstanding that regression models are reliable only within the data range they learned from.

#3 Including irrelevant or highly correlated variables without checking.

Wrong approach:

mdl = fitlm([x1 x2 x3], y); % Without checking variable relationships

Correct approach:

% Perform correlation analysis or feature selection before modeling

Root cause:

Assuming more data always improves models without understanding variable relationships.

Key Takeaways

Regression analysis finds the best mathematical relationship between inputs and an output to make predictions.

Visualizing data and understanding variables are essential first steps before fitting regression models.

Regression coefficients quantify how inputs affect outputs, but interpreting them requires care and context.

Model quality metrics and assumption checks are critical to trust regression results and avoid mistakes.

Advanced knowledge of assumptions, regularization, and diagnostics separates basic users from experts.

Practice

1.

What is the main purpose of regression analysis in MATLAB?

easy

A. To find relationships between variables

B. To sort data alphabetically

C. To create pie charts

D. To calculate matrix inverses

Solution

1. Step 1: Understand regression analysis goal

   Regression analysis is used to find how one variable depends on others.

2. Step 2: Match purpose with options

   Only To find relationships between variables describes finding relationships between variables, which is the goal of regression.

3. Final Answer:

   To find relationships between variables -> Option A

4. Quick Check:

   Regression = find relationships [OK]

Hint: Regression = relationship finding between numbers [OK]

Common Mistakes:

• Confusing regression with sorting or plotting
• Thinking regression calculates matrix inverses

2.

Which MATLAB command correctly fits a linear regression model to data X and response Y?

?

easy

A. mdl = fitlm(X, Y);

B. mdl = linearfit(X, Y);

C. mdl = regress(X, Y);

D. mdl = fitmodel(X, Y);

Solution

1. Step 1: Recall MATLAB function for linear regression

   The function fitlm fits linear regression models in MATLAB.

2. Step 2: Check options for correct syntax

   Only mdl = fitlm(X, Y); uses fitlm(X, Y) correctly; others are invalid or incorrect functions.

3. Final Answer:

   mdl = fitlm(X, Y); -> Option A

4. Quick Check:

   fitlm = linear model fit [OK]

Hint: Use fitlm for linear regression in MATLAB [OK]

Common Mistakes:

• Using non-existent functions like linearfit or fitmodel
• Confusing regress syntax with fitlm

3.

What is the output of this MATLAB code?

X = (1:5)';
Y = [2; 4; 6; 8; 10];
mdl = fitlm(X, Y);
disp(mdl.Coefficients.Estimate(2));

medium

A. 10

B. 2

C. 1

D. 4

Solution

1. Step 1: Understand data and model

   X is 1 to 5, Y is 2,4,6,8,10 which is Y = 2*X exactly.

2. Step 2: Interpret coefficient output

   The slope (coefficient for X) is 2, so mdl.Coefficients.Estimate(2) returns 2.

3. Final Answer:

   2 -> Option B

4. Quick Check:

   Slope = 2 [OK]

Hint: Slope equals change in Y over X [OK]

Common Mistakes:

• Confusing intercept with slope coefficient
• Misreading coefficient index

4.

Identify the error in this MATLAB code for regression:

X = [1 2 3 4 5];
Y = [2; 4; 6; 8; 10];
mdl = fitlm(X, Y);

medium

A. fitlm requires three input arguments

B. fitlm cannot handle numeric data

C. Y must be a matrix, not a vector

D. X and Y should be column vectors, not row vectors

Solution

1. Step 1: Check input shapes for fitlm

   fitlm expects predictors and response as column vectors or tables.

2. Step 2: Identify vector orientation

   X is a row vector and Y is a column vector; they should be column vectors for fitlm to work properly.

3. Final Answer:

   X and Y should be column vectors, not row vectors -> Option D

4. Quick Check:

   fitlm needs column vectors [OK]

Hint: Use column vectors for fitlm inputs [OK]

Common Mistakes:

• Passing row vectors instead of column vectors
• Thinking fitlm needs 3 inputs
• Believing fitlm can't handle numeric vectors

5.

You have data vectors X and Y with some outliers. Which approach in MATLAB regression helps reduce outlier impact?

hard

A. Remove all data points manually

B. Use fitlm(X, Y) without options

C. Use fitlm(X, Y, ''RobustOpts'', ''on'')

D. Use polyfit(X, Y, 2) for quadratic fit

Solution

1. Step 1: Understand outlier effect on regression

   Outliers can distort ordinary least squares regression results.

2. Step 2: Identify MATLAB option for robustness

   Using RobustOpts set to 'on' in fitlm applies robust regression reducing outlier influence.

3. Final Answer:

   Use fitlm(X, Y, ''RobustOpts'', ''on'') -> Option C

4. Quick Check:

   RobustOpts = reduce outlier impact [OK]

Hint: Add RobustOpts 'on' to fitlm for outliers [OK]

Common Mistakes:

• Ignoring outliers and using default fitlm
• Removing data points without analysis
• Using polynomial fit which may not reduce outliers