Bird
Raised Fist0

Optimizers - SGD, Adam, AdaGrad, Learning Rate Schedules

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/4
SGD Step Size

SGD uses a constant step size of 0.01 for all parameter updates.

🎯 Understanding the simplicity and limitations of constant learning rates in SGD.
learning_rate = 0.01
for param in params:
    param -= learning_rate * param.grad
Adam m, v terms

Adam computes moving averages of gradients (m) and squared gradients (v).

🎯 Recognizing how Adam adapts learning rates using momentum.
m = beta1 * m + (1 - beta1) * grad
v = beta2 * v + (1 - beta2) * (grad ** 2)
AdaGrad Accumulation

AdaGrad accumulates squared gradients, leading to a decrease in learning rate.

🎯 Understanding how AdaGrad adapts learning rates based on gradient history.
accum_grad += grad ** 2
adjusted_lr = learning_rate / (sqrt(accum_grad) + epsilon)
Learning Rate Decay

Learning rate schedules reduce the learning rate over time to ensure convergence.

🎯 Knowing when and why to apply learning rate decay in training.
decay_rate = 0.1
learning_rate *= (1.0 / (1.0 + decay_rate * epoch))
📊
Diagram
Diagram
┌───────────────────────────────┐
│       Optimizer Flow          │
├──────────────┬───────────────┤
│  SGD         │  Adam         │
│  ┌─────────┐ │  ┌─────────┐  │
│  │  Step   │ │  │  Step   │  │
│  │  Size   │ │  │  Size   │  │
│  └─────────┘ │  └─────────┘  │
│              │  ┌─────────┐  │
│              │  │  m, v   │  │
│              │  └─────────┘  │
├──────────────┼───────────────┤
│  AdaGrad     │  Learning Rate│
│  ┌─────────┐ │  Schedules    │
│  │  Accum  │ │  ┌─────────┐  │
│  │  Grad   │ │  │  Decay  │  │
│  └─────────┘ │  └─────────┘  │
└──────────────┴───────────────┘
Labels
  • SGD: Uses a fixed step size for updates
  • Adam: Combines step size with momentum terms m and v
  • AdaGrad: Accumulates squared gradients for adaptive learning rates
  • Learning Rate Schedules: Adjusts learning rate over time
🔑
Key Moments
▲
Q: Why does Adam use both m and v terms?
A: Adam uses m for momentum and v for scaling the learning rate, combining benefits of both SGD with momentum and RMSProp.
Q: How does AdaGrad's accumulation affect long-term training?
A: AdaGrad's accumulation can lead to overly small learning rates, slowing down training in the long term.
Q: What is the advantage of using learning rate schedules?
A: Learning rate schedules help in fine-tuning the convergence process by reducing the learning rate as training progresses, preventing overshooting.
Trace Summary

This document explains the mechanisms of various optimizers used in deep learning, including SGD, Adam, and AdaGrad, along with learning rate schedules. The diagram shows how each optimizer updates model parameters differently, highlighting the role of step size, momentum, and gradient accumulation. The trace steps provide a detailed walkthrough of how these optimizers function, with code snippets illustrating their implementation. Key moments focus on understanding the advantages and limitations of each optimizer, preparing learners for deeper discussions in interviews.

Primary Focus

✓ Understanding the mechanisms of different optimizers and learning rate schedules

Practice

(1/5)
1. In a deep learning model, when is it more appropriate to use Layer Normalization instead of Batch Normalization?
easy
A. When the model is trained on a very large batch size.
B. When the model is trained on a very small batch size or online learning.
C. When the model has a very deep architecture.
D. When the model is trained on a balanced dataset.

Solution

  1. Step 1: Understand the context of normalization techniques.

    Batch Normalization relies on batch statistics, which can be unstable with small batch sizes. Layer Normalization uses layer statistics, making it more stable for small batches or online learning.

  2. Step 2: Eliminate wrong options.

    When the model is trained on a very large batch size. is incorrect because large batch sizes are suitable for Batch Normalization. When the model has a very deep architecture. is incorrect because depth doesn't directly affect the choice between these normalizations. When the model is trained on a balanced dataset. is irrelevant to normalization choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Layer Norm for small batches or online learning. [OK]
Hint: Layer Norm for small batches or online learning. [OK]
Common Mistakes:
2. Given the following Python code snippet for a segmentation task, what will be the output shape if the input image shape is (256, 256, 3) and the model outputs a segmentation map with 10 classes?
import numpy as np
input_image = np.random.rand(256, 256, 3)
model_output = np.random.rand(256, 256, 10)
easy
A. (256, 256, 10)
B. (256, 256, 1)
C. (256, 256, 3)
D. (256, 256)

Solution

  1. Step 1: Understand the model output

    The model outputs a segmentation map with a shape corresponding to the input spatial dimensions and the number of classes.

  2. Step 2: Eliminate wrong options

    (256, 256, 3) is the input shape, (256, 256, 1) implies binary segmentation, and (256, 256) lacks class dimension.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Output shape = Input spatial + class channels [OK]
Hint: Output shape = Input spatial + class channels [OK]
Common Mistakes:
3. In logistic regression, what does the decision boundary represent?
easy
A. The point where the model's accuracy is maximized.
B. The point where the loss function is minimized.
C. The line that separates the data into two equal parts.
D. The threshold where the predicted probability is exactly 0.5.

Solution

  1. Step 1: Understand the decision boundary.

    The decision boundary in logistic regression is where the predicted probability is 0.5, indicating equal likelihood of either class.

  2. Step 2: Eliminate wrong options.

    The point where the model's accuracy is maximized. is incorrect because accuracy is not directly related to the decision boundary. The line that separates the data into two equal parts. is incorrect as it describes a median, not a decision boundary. The point where the loss function is minimized. is incorrect because the loss function is minimized during training, not at the decision boundary.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Decision boundary = probability 0.5 [OK]
Hint: Decision boundary = probability 0.5 [OK]
Common Mistakes:
4. In the context of the No Free Lunch Theorem, which statement is true about model performance across different datasets?
easy
A. A model that performs well on one dataset will perform well on all datasets.
B. No single model performs best on all possible datasets.
C. The best model for a dataset can be determined without testing.
D. A complex model always outperforms a simple model.

Solution

  1. Step 1: Understand the No Free Lunch Theorem

    The theorem states that no single model is universally best across all datasets.

  2. Step 2: Eliminate wrong options

    A model that performs well on one dataset will perform well on all datasets. is incorrect because model performance varies by dataset. The best model for a dataset can be determined without testing. is incorrect because testing is necessary to determine the best model. A complex model always outperforms a simple model. is incorrect because complexity does not guarantee better performance.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No universal best model. [OK]
Hint: No universal best model. [OK]
Common Mistakes:
5. How would you adapt K-Means to handle a dataset with categorical features?
hard
A. Use one-hot encoding and apply standard K-Means
B. Use K-Modes clustering instead
C. Convert categorical features to numerical using label encoding
D. Ignore categorical features and cluster on numerical features only

Solution

  1. Step 1: Understand K-Means limitations

    K-Means is not suitable for categorical data as it relies on Euclidean distance.

  2. Step 2: Choose the right adaptation

    K-Modes is designed for clustering categorical data, making it the best choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    K-Modes for categorical data [OK]
Hint: K-Modes for categorical data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard