Bird
Raised Fist0

Implement Linear Regression - Gradient Descent from Scratch

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/5
Initial Weights and Bias

Weights (w) initialized to 0.0, Bias (b) initialized to 0.0

🎯 Understanding initialization is crucial for gradient descent convergence.
w = 0.0
b = 0.0
Compute Predictions

Predictions for input [1, 2, 3] are [0.0, 0.0, 0.0] with initial weights and bias.

🎯 Initial predictions are zero due to zero initialization.
predictions = [w * x + b for x in [1, 2, 3]]
Calculate Loss

Loss calculated as 4.67 for predictions [0.0, 0.0, 0.0] against true values [1, 2, 3].

🎯 Loss function quantifies prediction error.
loss = sum((y - pred) ** 2 for y, pred in zip([1, 2, 3], predictions)) / 3
Compute Gradients

Gradient for weights computed as -4.0, for bias as -2.0.

🎯 Gradients indicate the direction to adjust weights and bias.
grad_w = -2 * sum((y - pred) * x for x, y, pred in zip([1, 2, 3], [1, 2, 3], predictions)) / 3
grad_b = -2 * sum(y - pred for y, pred in zip([1, 2, 3], predictions)) / 3
Update Weights and Bias

Weights updated to 0.4, Bias updated to 0.2 using learning rate 0.1.

🎯 Parameter update step is crucial for model learning.
w -= 0.1 * grad_w
b -= 0.1 * grad_b
📊
Diagram
Diagram
┌───────────────────────────────┐
│ Linear Regression Model       │
├───────────────────────────────┤
│ Initial Weights: w = 0.0      │
│ Initial Bias: b = 0.0         │
├───────────────────────────────┤
│ Step 1: Compute Predictions   │
│ Step 2: Calculate Loss        │
│ Step 3: Compute Gradients     │
│ Step 4: Update Weights & Bias │
└───────────────────────────────┘
Labels
  • Linear Regression Model: The model being trained
  • Initial Weights: Starting value of weights
  • Initial Bias: Starting value of bias
  • Compute Predictions: Calculate predicted values
  • Update Weights & Bias: Adjust parameters using gradients
🔑
Key Moments
▲
Q: Why is it important to initialize weights and bias to zero?
A: Zero initialization provides a neutral starting point, but care must be taken to avoid symmetry issues in more complex models.
Q: How does the learning rate affect the gradient descent process?
A: The learning rate determines the step size for each update; too large can overshoot, too small can slow convergence.
Q: Why do we compute gradients during training?
A: Gradients provide the direction and magnitude for updating model parameters to minimize the loss function.
Trace Summary

This visual trace document explains the process of implementing linear regression using gradient descent from scratch. The ASCII diagram outlines the key steps: initializing weights and bias, computing predictions, calculating loss, computing gradients, and updating parameters. The execution trace provides a step-by-step walkthrough of these processes, highlighting the importance of each step in the learning process. Key moments focus on understanding initialization, the impact of learning rate, and the role of gradients in training.

Primary Focus

✓ Implementing linear regression using gradient descent

Practice

(1/5)
1. What does the following Python code output when applied to an image array img? 
import numpy as np

def flip_image(img):
    return np.flip(img, axis=1)

img = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
flipped_img = flip_image(img)
print(flipped_img)
easy
A. [[3, 2, 1], [6, 5, 4], [9, 8, 7]]
B. [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
C. [[7, 8, 9], [4, 5, 6], [1, 2, 3]]
D. [[9, 8, 7], [6, 5, 4], [3, 2, 1]]

Solution

  1. Step 1: Understand the flip operation

    The np.flip function flips the array along the specified axis. Here, axis=1 means flipping horizontally.

  2. Step 2: Apply the flip to the array

    Flipping horizontally changes the order of elements in each row.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Horizontal flip reverses row elements. [OK]
Hint: Horizontal flip reverses row elements. [OK]
Common Mistakes:
2. A Random Forest implementation is producing poor results. The trees seem to be too similar. What is the most likely cause?
medium
A. Insufficient feature sampling
B. High variance in the data
C. Too much data in each bootstrap sample
D. Too many trees in the forest

Solution

  1. Step 1: Identify the Cause of Similar Trees

    Similar trees often result from insufficient randomness in feature selection.

  2. Step 2: Eliminate Other Causes

    Too many trees (A) or high variance (D) don't cause similarity; too much data (C) affects overfitting, not similarity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feature sampling adds diversity [OK]
Hint: Feature sampling adds diversity [OK]
Common Mistakes:
3. How can you optimize the following data augmentation pipeline for memory efficiency? 
def augment_image(img):
    img = np.flip(img, axis=1)
    img = img[10:110, 10:110]
    img = (img - np.mean(img)) / np.std(img)
    return img
hard
A. Chain operations without intermediate assignments.
B. Pre-compute mean and std for normalization.
C. Use a smaller dtype for the image array.
D. Use in-place operations for flipping and cropping.

Solution

  1. Step 1: Identify memory inefficiencies

    Intermediate assignments create unnecessary copies.

  2. Step 2: Evaluate optimization strategies

    Chaining operations reduces memory usage by avoiding intermediate arrays.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Chain operations to reduce memory usage. [OK]
Hint: Chain operations to reduce memory usage. [OK]
Common Mistakes:
4. How can you optimize the memory usage of a Min-Max scaling implementation for a large dataset?
hard
A. Process data in chunks instead of loading all at once
B. Use float64 data type for higher precision
C. Use a single loop to calculate min and max
D. Store scaled data in a new array to avoid overwriting

Solution

  1. Step 1: Identify Memory Bottlenecks

    Loading large datasets entirely into memory can be inefficient.

  2. Step 2: Optimize Memory Usage

    Processing data in chunks reduces memory footprint, making it feasible to handle large datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Chunk processing saves memory. [OK]
Hint: Chunk processing saves memory. [OK]
Common Mistakes:
5. How can you optimize the following non-max suppression code for large datasets? 
def non_max_suppression(boxes, scores, iou_threshold):
    indices = np.argsort(scores)[::-1]
    keep = []
    while len(indices) > 0:
        current = indices[0]
        keep.append(current)
        indices = indices[1:]
        indices = [i for i in indices if iou(boxes[current], boxes[i]) < iou_threshold]
    return keep
hard
A. Parallelize the IoU calculations.
B. Use a more efficient sorting algorithm.
C. Reduce the IoU threshold.
D. Use a different data structure for indices.

Solution

  1. Step 1: Identify bottleneck

    IoU calculations are the most computationally expensive part.

  2. Step 2: Optimize bottleneck

    Parallelizing IoU calculations can significantly reduce runtime for large datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Parallelize expensive operations [OK]
Hint: Parallelize expensive operations [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard