Activation functions (ReLU, Sigmoid, Softmax) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of an activation function in a neural network?

An activation function adds non-linearity to the model, allowing it to learn complex patterns and make decisions beyond simple linear relationships.

beginner

Describe the ReLU activation function and its output behavior.

ReLU (Rectified Linear Unit) outputs the input directly if it is positive; otherwise, it outputs zero. It helps models learn faster and reduces the chance of vanishing gradients.

beginner

What is the range of the Sigmoid activation function and why is it useful?

Sigmoid outputs values between 0 and 1, making it useful for models that predict probabilities or binary outcomes.

intermediate

Explain the Softmax activation function and when it is typically used.

Softmax converts a vector of numbers into probabilities that sum to 1. It is commonly used in the output layer for multi-class classification problems.

intermediate

What problem does ReLU help to solve compared to Sigmoid or Tanh?

ReLU helps reduce the vanishing gradient problem by allowing gradients to flow when inputs are positive, unlike Sigmoid or Tanh which can squash gradients to near zero.

Which activation function outputs values strictly between 0 and 1?

A Softmax

B ReLU

C Linear

D Sigmoid

Sigmoid outputs values between 0 and 1, making it suitable for probability predictions.

What does the ReLU function output when the input is negative?

A The input value

B Negative of the input

C Zero

D One

ReLU outputs zero for any negative input.

Softmax is mainly used in which part of a neural network?

A Output layer for multi-class classification

B Hidden layers

C Input layer

D Output layer for regression

Softmax converts outputs into probabilities for multi-class classification in the output layer.

Which activation function can cause the vanishing gradient problem?

A Sigmoid

B ReLU

C Softmax

D Leaky ReLU

Sigmoid can squash gradients to near zero, causing vanishing gradients.

What is a key benefit of using ReLU over Sigmoid?

A Outputs probabilities

B Non-linear but faster to compute and reduces vanishing gradients

C Always outputs positive values

D Used for multi-class classification

ReLU is simple, fast, and helps reduce vanishing gradients compared to Sigmoid.

Explain how ReLU, Sigmoid, and Softmax activation functions differ in their output and typical use cases.

Think about the output range and where in the network each is used.

Describe why activation functions are important in neural networks and what would happen without them.

Consider what linear vs non-linear means for learning.