Bird
Raised Fist0

Linear Regression - From Least Squares to Gradient Descent

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Least Squares vs Gradient Descent

When training a linear regression model on a dataset with 1 million samples and 100 features, would you use least squares or gradient descent? Justify your choice.

📊
Comparison
DimensionLeast SquaresGradient Descent
Training speedSlow for large nFast for large n
Memory usageHighLow
ScalabilityLowHigh
Solution precisionExactApproximate
Sensitivity to feature scalingLowHigh
✅
Verdict
Default to gradient descent for datasets with n > 100,000 due to its scalability and memory efficiency. Use least squares when n < 10,000 and an exact solution is needed, provided the dataset fits in memory.
💡
Key Insight
Choose least squares for exact solutions on small datasets and gradient descent for scalability on large datasets.
⚠️
Interview Trap
❌ Wrong: Always use gradient descent for large datasets.
Ignoring the fact that least squares can be more efficient for small datasets that fit in memory.
✅ Consider the dataset size and memory constraints; use least squares for small, well-conditioned datasets and gradient descent for large-scale problems.
Least Squares
"Provides an exact solution for linear regression when the dataset is small and well-conditioned."
Weaknesses
  • Computationally expensive for large datasets (O(n^2) complexity).
  • Requires the entire dataset to fit in memory.
  • Sensitive to multicollinearity and outliers.
  • Not suitable for online learning.
Ideal When
  • n < 10,000
  • Features are well-conditioned (no multicollinearity).
  • Dataset fits in memory.
  • Low noise in data.
Small-scale academic research projects often use least squares for its simplicity and exactness.
Gradient Descent
"Scales well with large datasets and can handle online learning."
Weaknesses
  • Requires careful tuning of hyperparameters like learning rate.
  • Convergence can be slow without optimization techniques.
  • Provides an approximate solution.
  • Sensitive to feature scaling.
Ideal When
  • n > 100,000
  • Limited memory resources.
  • Need for online or incremental learning.
  • High-dimensional feature space.
Google uses gradient descent for training large-scale models where data is too large to fit in memory.
🐍
Least Squares - Example
▼ Show
Python
import numpy as np
X = np.random.rand(100, 10)
y = np.random.rand(100)
coefficients = np.linalg.inv(X.T @ X) @ X.T @ y
print(coefficients)
🐍
Gradient Descent - Example
▼ Show
Python
import numpy as np
X = np.random.rand(1000000, 100)
y = np.random.rand(1000000)
weights = np.zeros(100)
learning_rate = 0.01
for _ in range(100):
    predictions = X @ weights
    gradient = X.T @ (predictions - y) / len(y)
    weights -= learning_rate * gradient
print(weights)
✅
Use Least Squares When
  • •n < 10,000
  • •Dataset fits entirely in memory
  • •Exact solution is required
  • •Low feature count (d < 100)
✅
Use Gradient Descent When
  • •n > 100,000
  • •Memory constraints exist
  • •Online learning is needed
  • •High-dimensional data (d > 100)
↳
Nuances
  • For small datasets with multicollinearity, least squares can give misleading results due to high variance.
  • Gradient descent can be inefficient if the learning rate is not properly tuned, leading to slow convergence.
  • In high-dimensional spaces, gradient descent benefits significantly from feature scaling, unlike least squares.
  • Least squares can be more interpretable due to its closed-form solution, which is easier to explain.

Practice

(1/5)
1. In the context of data collection for machine learning, which scenario best describes an offline data collection approach?
easy
A. Recording user feedback immediately after a transaction.
B. Collecting user interaction data in real-time as they browse a website.
C. Using sensors to monitor environmental conditions continuously.
D. Gathering historical sales data from a database for analysis.

Solution

  1. Step 1: Identify offline data collection characteristics

    Offline data collection involves gathering data that has already been generated and stored, not in real-time.

  2. Step 2: Eliminate real-time options

    Options A, C, and D involve real-time or near-real-time data collection, which is characteristic of online data collection.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Offline = historical, stored data. [OK]
Hint: Offline = historical, stored data. [OK]
Common Mistakes:
2. A model using Batch Normalization is showing unstable training behavior with fluctuating loss. What is the most likely cause?
medium
A. The batch size is too small, leading to unreliable batch statistics.
B. The learning rate is too high, causing instability.
C. The model architecture is too deep, causing vanishing gradients.
D. The dataset is imbalanced, affecting the normalization process.

Solution

  1. Step 1: Identify the role of batch size in Batch Normalization.

    Batch Normalization relies on accurate batch statistics, which can be unreliable with small batch sizes.

  2. Step 2: Eliminate other causes.

    The learning rate is too high, causing instability. is a common issue but not specific to Batch Norm. The model architecture is too deep, causing vanishing gradients. is unrelated to normalization. The dataset is imbalanced, affecting the normalization process. affects model performance but not directly related to Batch Norm instability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Small batch size causes unstable Batch Norm. [OK]
Hint: Small batch size causes unstable Batch Norm. [OK]
Common Mistakes:
3. For a real-time video surveillance system, which object detection model would be more suitable and why?
medium
A. R-CNN, because it provides higher accuracy
B. YOLO, because it is more robust to occlusions
C. YOLO, because it processes frames faster
D. R-CNN, because it handles multiple objects better

Solution

  1. Step 1: Identify system requirements

    Real-time systems prioritize speed over marginal accuracy improvements.

  2. Step 2: Match model to requirements

    YOLO is designed for fast processing, making it suitable for real-time applications.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Real-time = Speed, choose YOLO [OK]
Hint: Real-time = Speed, choose YOLO [OK]
Common Mistakes:
4. A model performs well overall but poorly on a specific slice of data. What is the most likely cause?
medium
A. The slice has a different distribution than the training data.
B. The model has too many parameters.
C. The model is overfitting to the training data.
D. The learning rate is too high.

Solution

  1. Step 1: Analyze the problem

    Poor performance on a specific slice often indicates distribution mismatch.

  2. Step 2: Eliminate wrong options

    The model is overfitting to the training data. is incorrect as overfitting would affect overall performance. The model has too many parameters. is unrelated to slice-specific issues. The learning rate is too high. affects convergence, not slice performance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Slice mismatch = distribution issue. [OK]
Hint: Slice mismatch = distribution issue. [OK]
Common Mistakes:
5. A language model using SentencePiece is producing unexpected token sequences. What is a likely cause?
medium
A. The SentencePiece model was trained without proper normalization.
B. The training data was too diverse.
C. The model's vocabulary size is too large.
D. The SentencePiece model was trained with a fixed vocabulary size.

Solution

  1. Step 1: Identify potential issues with SentencePiece training.

    Improper normalization can lead to unexpected token sequences.

  2. Step 2: Eliminate wrong options.

    Large vocabulary (A) and diverse data (C) are less likely to cause unexpected sequences; fixed vocabulary (D) is unrelated to sequence issues.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Normalization issues cause token surprises. [OK]
Hint: Normalization issues cause token surprises. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard