Bird
Raised Fist0

Activation Functions - ReLU, Sigmoid, GELU, Why ReLU Dominates

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
ReLU vs Sigmoid

In a deep learning model for image classification with millions of parameters, would you choose ReLU or Sigmoid as the activation function? Justify your choice.

📊
Comparison
DimensionReLUSigmoid
Training SpeedFastSlow
Convergence RateHighModerate
InterpretabilityLowHigh
Handling of Large ModelsExcellentPoor
Gradient StabilityModerateLow
✅
Verdict
Default to ReLU for deep networks with large datasets and sufficient compute resources. Switch to Sigmoid when interpretability as probability is crucial and computational resources are limited.
💡
Key Insight
ReLU dominates in deep networks due to its computational efficiency and sparse activation, but Sigmoid remains valuable for its probabilistic interpretation in simpler models.
⚠️
Interview Trap
❌ Wrong: Always use ReLU because it's faster.
Ignoring the vanishing gradient problem in deep networks where Sigmoid might be more stable.
✅ Consider the model depth and computational resources; ReLU is preferred for deep networks with sufficient compute, while Sigmoid is better for shallow networks needing probabilistic outputs.
ReLU
"Efficient computation and sparse activation leading to faster convergence."
Weaknesses
  • Dying ReLU problem where neurons can become inactive.
  • Not zero-centered, which can slow convergence.
  • Sensitive to outliers, leading to exploding gradients.
Ideal When
  • n > 100K samples and model depth > 10 layers.
  • When using GPUs for training.
  • When training time is limited to < 24 hours.
Google's Inception models use ReLU due to its efficiency in deep architectures.
Sigmoid
"Smooth gradient and probabilistic interpretation for binary classification."
Weaknesses
  • Vanishing gradient problem in deep networks.
  • Computationally expensive due to exponential function.
  • Output not zero-centered, affecting convergence speed.
Ideal When
  • n < 10K samples and model depth < 5 layers.
  • When interpretability of output as probability is crucial.
  • When using CPUs with limited computational resources.
Early neural networks like the original perceptron used Sigmoid for binary classification tasks.
🐍
ReLU - Example
▼ Show
Python
import numpy as np

def relu(x):
    return np.maximum(0, x)

x = np.array([-2, -1, 0, 1, 2])
output = relu(x)
print(output)  # Output: [0 0 0 1 2]
🐍
Sigmoid - Example
▼ Show
Python
import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

x = np.array([-2, -1, 0, 1, 2])
output = sigmoid(x)
print(output)  # Output: [0.11920292 0.26894142 0.5 0.73105858 0.88079708]
✅
Use ReLU When
  • •n > 100K and model depth > 10 layers.
  • •When using GPUs and training time < 24 hours.
  • •When sparse activation is beneficial for speed.
✅
Use Sigmoid When
  • •n < 10K and model depth < 5 layers.
  • •When interpretability as probability is needed.
  • •When using CPUs with limited resources.
↳
Nuances
  • ReLU can suffer from dying neurons, especially in shallow networks.
  • Sigmoid's probabilistic output is useful in binary classification but not in deep layers.
  • ReLU's non-zero-centered output can slow convergence if not handled properly.
  • Sigmoid's vanishing gradient can be mitigated with careful initialization and architecture design.

Practice

(1/5)
1. In the context of Naive Bayes, which of the following best describes the assumption of conditional independence?
easy
A. Features are independent of the class label.
B. Features are independent given the class label.
C. Features are dependent on each other given the class label.
D. Features are independent in all scenarios.

Solution

  1. Step 1: Understand Conditional Independence

    Naive Bayes assumes that features are independent given the class label.

  2. Step 2: Eliminate Wrong Options

    Features are independent of the class label. suggests features are independent of the class label, which is incorrect. Features are dependent on each other given the class label. suggests dependency, which contradicts the assumption. Features are independent in all scenarios. suggests independence in all scenarios, which is overly broad.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Naive Bayes assumes independence given class. [OK]
Hint: Naive Bayes assumes independence given class. [OK]
Common Mistakes:
2. 
import numpy as np
from sklearn.cluster import KMeans

X = np.array([[1, 2], [1, 4], [1, 0],
              [4, 2], [4, 4], [4, 0]])
kmeans = KMeans(n_clusters=2, random_state=0).fit(X)
print(kmeans.labels_)
What does the code output?
easy
A. [0 0 0 1 1 1]
B. [1 1 1 0 0 0]
C. [0 1 0 1 0 1]
D. [1 0 1 0 1 0]

Solution

  1. Step 1: Understand KMeans clustering.

    KMeans assigns labels based on proximity to cluster centers.

  2. Step 2: Analyze the data points.

    Points [1, 2], [1, 4], [1, 0] are closer to one center, and [4, 2], [4, 4], [4, 0] to another.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cluster proximity determines labels [OK]
Hint: Cluster proximity determines labels [OK]
Common Mistakes:
3. A model trained with Adam optimizer is not converging. What is the most likely cause?
medium
A. Batch size is too large
B. Learning rate is too high
C. Adam is not suitable for non-convex problems
D. Weight decay is too low

Solution

  1. Step 1: Identify common issues with Adam.

    Adam can be sensitive to high learning rates, causing divergence.

  2. Step 2: Eliminate less likely causes.

    Adam handles non-convex problems well, and batch size/weight decay are less likely to cause non-convergence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High lr can cause Adam to diverge. [check]
Hint: High lr can cause Adam to diverge. [check]
Common Mistakes:
4. A model trained on a dataset shows unexpected bias against a minority group. What is the most likely root cause?
medium
A. The training data is imbalanced.
B. The learning rate is too high.
C. The model has too many parameters.
D. The model uses a linear activation function.

Solution

  1. Step 1: Identify the symptom.

    Bias against a minority group indicates potential data imbalance.

  2. Step 2: Analyze the options.

    Imbalanced data (B) is a common cause of bias. Other options relate to model complexity or training dynamics, not bias directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Imbalanced data often leads to bias. [OK]
Hint: Imbalanced data often leads to bias. [OK]
Common Mistakes:
5. When choosing between BPE and WordPiece for a language model, which scenario makes WordPiece a better choice?
medium
A. When the vocabulary size needs to be minimized.
B. When handling a language with a high degree of inflection.
C. When the model needs to handle rare words efficiently.
D. When the model is primarily used for text generation tasks.

Solution

  1. Step 1: Identify the strengths of WordPiece.

    WordPiece is designed to handle rare words efficiently by breaking them into subwords.

  2. Step 2: Eliminate wrong options.

    Minimizing vocabulary size (A) and handling inflection (B) are not specific strengths of WordPiece; text generation (D) is not a specific advantage.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WordPiece excels with rare words. [OK]
Hint: WordPiece excels with rare words. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard