Bird
Raised Fist0

Optimizers - SGD, Adam, AdaGrad, Learning Rate Schedules

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
SGD with Learning Rate Schedules vs Adam

For training a deep neural network with limited compute resources, would you choose SGD with learning rate schedules or Adam? Justify your choice.

📊
Comparison
DimensionSGD with Learning Rate SchedulesAdam
Convergence SpeedMediumHigh
Memory UsageLowHigh
Hyperparameter SensitivityHighLow
ScalabilityHighMedium
Adaptability to Non-Stationary DataLowHigh
✅
Verdict
Default to Adam for small to medium datasets with abundant compute resources and quick prototyping needs. Switch to SGD with learning rate schedules for large-scale datasets where compute efficiency and scalability are critical.
💡
Key Insight
The choice between SGD with learning rate schedules and Adam hinges on dataset size and compute resource availability.
⚠️
Interview Trap
❌ Wrong: Always use Adam because it converges faster.
Ignoring the scalability and memory efficiency of SGD with learning rate schedules on large datasets.
✅ Consider the dataset size and compute resources; SGD with learning rate schedules is preferable for large-scale problems with limited compute.
SGD with Learning Rate Schedules
"Highly efficient with fine-tuned learning rate schedules for large-scale data."
Weaknesses
  • Requires careful tuning of learning rate schedules.
  • Slower convergence on complex, non-convex problems.
  • Sensitive to initial learning rate choice.
Ideal When
  • n > 100K and compute resources are limited.
  • Training time > 24 hours available for fine-tuning.
  • When model interpretability is crucial.
Google uses SGD with learning rate schedules for training large-scale language models where compute efficiency is critical.
Adam
"Fast convergence with minimal hyperparameter tuning."
Weaknesses
  • Higher memory usage due to additional parameters.
  • Can overfit on small datasets.
  • Less effective on very large datasets without tuning.
Ideal When
  • n < 100K and compute resources are abundant.
  • Quick prototyping with limited tuning time.
  • When model needs to adapt to non-stationary data.
OpenAI uses Adam for training models where rapid convergence is necessary, such as in reinforcement learning tasks.
🐍
SGD with Learning Rate Schedules - Example
▼ Show
Python
import numpy as np
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.callbacks import LearningRateScheduler

model.compile(optimizer=SGD(learning_rate=0.01, momentum=0.9),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Learning rate schedule
def lr_schedule(epoch):
    return 0.01 * (0.1 ** (epoch // 10))

model.fit(x_train, y_train, epochs=50, callbacks=[LearningRateScheduler(lr_schedule)])
🐍
Adam - Example
▼ Show
Python
import numpy as np
from tensorflow.keras.optimizers import Adam

model.compile(optimizer=Adam(learning_rate=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=50)
✅
Use SGD with Learning Rate Schedules When
  • •n > 100K and compute resources are limited.
  • •Training time > 24 hours available for fine-tuning.
  • •When model interpretability is crucial.
✅
Use Adam When
  • •n < 100K and compute resources are abundant.
  • •Quick prototyping with limited tuning time.
  • •When model needs to adapt to non-stationary data.
↳
Nuances
  • SGD with learning rate schedules can outperform Adam on very large datasets due to better scalability.
  • Adam's fast convergence can lead to overfitting on small datasets if not carefully monitored.
  • SGD requires more hyperparameter tuning but can achieve better generalization with the right schedule.

Practice

(1/5)
1. 
import numpy as np
from sklearn.preprocessing import StandardScaler

X = np.array([[1, 2], [3, 4], [5, 6]])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled.mean(axis=0))
What does this code output?
easy
A. [1. 1.]
B. [0.5 0.5]
C. [0. 0.]
D. [0. 1.]

Solution

  1. Step 1: Understand standardization.

    StandardScaler standardizes features by removing the mean and scaling to unit variance.

  2. Step 2: Calculate the mean of standardized data.

    After standardization, the mean of each feature should be 0.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    StandardScaler centers data to mean 0. [OK]
Hint: StandardScaler centers data to mean 0. [OK]
Common Mistakes:
2. In the context of evaluating search engine results, which metric is most appropriate for measuring the quality of the top-ranked results?
easy
A. Mean Reciprocal Rank (MRR)
B. Mean Average Precision (MAP)
C. Normalized Discounted Cumulative Gain (NDCG)
D. Precision at K

Solution

  1. Step 1: Understand the focus of each metric

    MRR is specifically designed to evaluate the rank of the first relevant result, making it ideal for top-ranked results.

  2. Step 2: Eliminate wrong options

    MAP considers all relevant documents, NDCG focuses on the entire ranking, and Precision at K is not specific to the first relevant result.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    MRR focuses on the first relevant result. [OK]
Hint: MRR focuses on the first relevant result. [OK]
Common Mistakes:
3. In the context of statistical significance testing in ML, which of the following best describes a p-value?
easy
A. The probability of observing the data given that the null hypothesis is true.
B. The probability that the null hypothesis is true.
C. The probability of the alternative hypothesis being true.
D. The probability of making a Type I error.

Solution

  1. Step 1: Understand p-value definition

    A p-value is the probability of observing the data, or something more extreme, given that the null hypothesis is true.

  2. Step 2: Eliminate wrong options

    The probability that the null hypothesis is true. is incorrect because p-value is not the probability that the null hypothesis is true. The probability of the alternative hypothesis being true. is incorrect because it does not relate to the alternative hypothesis. The probability of making a Type I error. is incorrect because it confuses p-value with Type I error probability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    P-value is about data given null hypothesis [OK]
Hint: P-value is about data given null hypothesis [OK]
Common Mistakes:
4. In a deep neural network, why might you choose GELU over ReLU?
medium
A. GELU is computationally cheaper than ReLU.
B. GELU is more interpretable than ReLU.
C. GELU avoids the dying neuron problem entirely.
D. GELU provides smoother gradients, which can improve convergence.

Solution

  1. Step 1: Compare GELU and ReLU characteristics.

    GELU provides smoother gradients due to its probabilistic nature, which can help in convergence.

  2. Step 2: Evaluate each option.

    GELU is not cheaper (A), doesn't entirely avoid dying neurons (C), and isn't more interpretable (D).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    GELU smooths gradients, aids convergence. [OK]
Hint: GELU smooths gradients, aids convergence. [OK]
Common Mistakes:
5. A model trained with K-Fold cross-validation is underperforming on unseen data. What is a likely cause?
medium
A. The model is overfitting due to too many folds.
B. Data leakage between training and validation sets.
C. The model is underfitting due to too few folds.
D. The model is not complex enough for the data.

Solution

  1. Step 1: Consider data leakage as a common issue.

    Data leakage can occur if the same data points appear in both training and validation sets.

  2. Step 2: Eliminate wrong options.

    Too many folds (A) typically doesn't cause overfitting, too few folds (C) would not cause underperformance on unseen data, model complexity (D) is unrelated to cross-validation setup.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Check for data leakage in CV. [OK]
Hint: Check for data leakage in CV. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard