Activation functions in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is an activation function in a neural network?

An activation function decides if a neuron should be activated or not. It adds non-linearity to the network, helping it learn complex patterns.

beginner

Why do we need non-linear activation functions?

Non-linear activation functions let neural networks learn and model complex data. Without them, the network would behave like a simple linear model.

beginner

Name three common activation functions used in neural networks.

Three common activation functions are: Sigmoid, ReLU (Rectified Linear Unit), and Tanh (Hyperbolic Tangent).

beginner

What is the output range of the Sigmoid activation function?

The Sigmoid function outputs values between 0 and 1, making it useful for probabilities.

beginner

How does the ReLU activation function work?

ReLU outputs the input directly if it is positive; otherwise, it outputs zero. This helps the network learn faster and reduces the chance of vanishing gradients.

What does an activation function add to a neural network?

A More layers

B Non-linearity

C Data normalization

D Weight initialization

Activation functions add non-linearity, allowing the network to learn complex patterns.

Which activation function outputs values between -1 and 1?

A ReLU

B Sigmoid

C Tanh

D Softmax

Tanh outputs values between -1 and 1, centering data around zero.

What is a key advantage of ReLU over Sigmoid?

A ReLU outputs probabilities

B ReLU is non-differentiable

C ReLU outputs values between 0 and 1

D ReLU helps reduce vanishing gradients

ReLU helps reduce the vanishing gradient problem, allowing faster training.

Which activation function is best for the output layer in binary classification?

A Sigmoid

B Tanh

C ReLU

D Linear

Sigmoid outputs values between 0 and 1, suitable for binary classification probabilities.

What happens to negative inputs in the ReLU function?

A They become zero

B They are passed through unchanged

C They become one

D They become negative one

ReLU outputs zero for any negative input.

Explain what an activation function is and why it is important in neural networks.

Think about how neurons decide to 'fire' or not.

Describe the differences between Sigmoid, Tanh, and ReLU activation functions.

Consider their output values and where they are commonly used.