Bird
Raised Fist0

Linear Regression - From Least Squares to Gradient Descent

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Linear Regression - From Least Squares to Gradient Descent
Understanding linear regression through least squares and gradient descent
theorybeginnerFreq: very_highClassical ML
Prerequisites:Why Machine Learning Exists
💡
Why It Exists

Linear regression was invented to solve the problem of predicting a continuous outcome based on one or more input features. It provides a simple yet powerful way to model the relationship between variables, allowing us to make predictions and understand the influence of each feature.

📖
What It Is

Linear regression is a method for modeling the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data. Imagine it as drawing the best straight line through a scatter plot of data points.

∑
Math Intuition
The least squares method minimizes the sum of the squared differences between the observed and predicted values. This means it penalizes larger errors more heavily, ensuring the model fits the data as closely as possible.
⚙️
Core Mechanism

Linear regression finds the line that minimizes the sum of squared differences between the observed values and the values predicted by the line.

📋
Step by Step
  1. Initialize the parameters (weights) of the model.
  2. Calculate the predicted values using the current parameters.
  3. Compute the error as the difference between predicted and actual values.
  4. Update the parameters to reduce the error using an optimization method like gradient descent.
  5. Repeat the process until the error is minimized or a stopping criterion is met.
🎯
Intuition
Think of linear regression as trying to find the flattest path down a hill (error surface) to reach the lowest point (minimum error).
🐍
Python Example
Python
import numpy as np

# Sample data
X = np.array([[1], [2], [3]])
y = np.array([2, 2.5, 3.5])

# Initialize weights
w = np.zeros((1,))

# Gradient Descent
learning_rate = 0.01
for _ in range(1000):
    predictions = X.dot(w)
    errors = predictions - y
    gradient = X.T.dot(errors) / len(y)
    w -= learning_rate * gradient

print(w)  # Expected output: array close to [0.5]
✅
When It Works
  • When the relationship between the independent and dependent variables is approximately linear.
  • When the dataset is not too large, allowing for efficient computation of the closed-form solution.
  • When the noise in the data is normally distributed and homoscedastic (constant variance).
⚠️
When It Breaks
  • Fails with multicollinearity, where independent variables are highly correlated, leading to unstable coefficient estimates.
  • Struggles with outliers, as they can disproportionately affect the model due to the squared error term.
  • Ineffective when the relationship between variables is non-linear, as it cannot capture complex patterns.
⇄
vs Polynomial Regression
Use This When
The relationship between variables is linear and the dataset is small to medium-sized.
Use Alternative When
The relationship is non-linear and you need to capture more complex patterns.
🎯
Interview Angles
  • Interviewer asks you to implement linear regression from scratch using gradient descent → testing your understanding of optimization and iterative improvement.
  • Interviewer presents a dataset with multicollinearity and asks how you would address it → assessing your knowledge of feature engineering and regularization.
  • Interviewer asks why your linear regression model is performing poorly on a dataset with outliers → evaluating your ability to diagnose model sensitivity issues.
⚠️
Common Mistakes
Believing linear regression can handle non-linear relationships without transformation, leading to poor model fit.
Assuming the model is robust to outliers, which can skew results significantly.
Thinking that adding more features will always improve the model, ignoring the risk of overfitting and multicollinearity.
📌
Summary
  • Linear regression models the relationship between variables using a straight line.
  • It minimizes the sum of squared errors to find the best-fitting line.
  • Be cautious of multicollinearity and outliers, which can destabilize the model.

Practice

(1/5)
1. In a deep learning model, when is it more appropriate to use Layer Normalization instead of Batch Normalization?
easy
A. When the model is trained on a very large batch size.
B. When the model is trained on a very small batch size or online learning.
C. When the model has a very deep architecture.
D. When the model is trained on a balanced dataset.

Solution

  1. Step 1: Understand the context of normalization techniques.

    Batch Normalization relies on batch statistics, which can be unstable with small batch sizes. Layer Normalization uses layer statistics, making it more stable for small batches or online learning.

  2. Step 2: Eliminate wrong options.

    When the model is trained on a very large batch size. is incorrect because large batch sizes are suitable for Batch Normalization. When the model has a very deep architecture. is incorrect because depth doesn't directly affect the choice between these normalizations. When the model is trained on a balanced dataset. is irrelevant to normalization choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Layer Norm for small batches or online learning. [OK]
Hint: Layer Norm for small batches or online learning. [OK]
Common Mistakes:
2. Given the following Python code using Word2Vec, what is the output shape of the word vector for 'king'? 
from gensim.models import Word2Vec
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, workers=4)
vector = model.wv['king']
easy
A. (100, 1)
B. (1, 100)
C. (100,)
D. (1,)

Solution

  1. Step 1: Understand Word2Vec output

    The Word2Vec model returns a 1D array of the specified vector size for each word.

  2. Step 2: Determine the shape

    The vector size is set to 100, so the output shape is (100,).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Word2Vec outputs 1D vector of size [OK]
Hint: Word2Vec outputs 1D vector of size [OK]
Common Mistakes:
3. In the context of LSTM networks, which component is primarily responsible for deciding what information to keep or discard from the cell state?
easy
A. The forget gate
B. The input gate
C. The output gate
D. The activation function

Solution

  1. Step 1: Identify the role of each gate in LSTM.

    The forget gate determines what information to discard from the cell state.

  2. Step 2: Eliminate wrong options.

    The input gate decides what new information to store, the output gate decides what to output, and the activation function is not a gate.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Forget gate discards info in LSTM. [OK]
Hint: Forget gate discards info in LSTM. [OK]
Common Mistakes:
4. Given the following Python code snippet, what will be the output if the model is trained with biased data?
from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 0, 0]

accuracy = accuracy_score(y_true, y_pred)
print(accuracy)
easy
A. 0.4
B. 0.8
C. 0.6
D. 0.2

Solution

  1. Step 1: Calculate the accuracy.

    Accuracy is the ratio of correctly predicted instances to the total instances. Here, 3 out of 5 predictions are correct.

  2. Step 2: Compute the accuracy.

    Correct predictions: [0, 1, 1] -> 0.6

  3. Final Answer:

    0.6 -> 0.6

  4. Quick Check:

    Count correct predictions, divide by total. [OK]
Hint: Count correct predictions, divide by total. [OK]
Common Mistakes:
5. When should you avoid using a polynomial kernel in SVM?
medium
A. When the dataset is very large
B. When the data has non-linear patterns
C. When the data is linearly separable
D. When you have a small number of features

Solution

  1. Step 1: Consider the computational cost of polynomial kernels

    Polynomial kernels can be computationally expensive, especially with large datasets.

  2. Step 2: Eliminate wrong options

    When the data is linearly separable is incorrect because a polynomial kernel can still be used but is unnecessary. When the data has non-linear patterns is incorrect because polynomial kernels are suitable for non-linear patterns. When you have a small number of features is incorrect because the number of features doesn't directly affect the choice of kernel.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Avoid polynomial kernel for large datasets [OK]
Hint: Avoid polynomial kernel for large datasets [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard