Activation functions in ML Python - Deep Dive

Overview - Activation functions

What is it?

Activation functions are simple mathematical formulas used in artificial neurons to decide if a neuron should be activated or not. They take the input signals, apply a transformation, and produce an output that helps the neural network learn complex patterns. Without activation functions, neural networks would behave like simple linear models and could not solve complicated problems. They add non-linearity, allowing networks to understand and model real-world data better.

Why it matters

Activation functions exist because real-world data and problems are rarely simple or straight lines. Without them, neural networks would only be able to solve very basic tasks, like drawing straight lines between points. This would make technologies like voice recognition, image understanding, and language translation impossible or very poor. Activation functions enable machines to learn and make decisions that feel intelligent and flexible.

Where it fits

Before learning activation functions, you should understand what neurons and layers are in neural networks. After mastering activation functions, you can explore how different network architectures use them and how to train networks effectively using backpropagation and optimization.

Mental Model

Core Idea

Activation functions decide how much signal passes through a neuron, adding the crucial non-linear twist that lets neural networks learn complex patterns.

Think of it like...

Activation functions are like the volume knob on a radio: they control how loud the signal is passed on, sometimes turning it off completely or boosting it, shaping what you finally hear.

Input Signal ──▶ [Activation Function] ──▶ Output Signal

Where the activation function can be:
  - Off (output zero)
  - Pass through (output same or scaled)
  - Squash (limit output between bounds)

This transforms the input before passing it forward.

Build-Up - 7 Steps

1

Foundation

What is an Activation Function?

Concept: Activation functions transform the input signal of a neuron to decide its output.

Imagine a neuron receiving a number. The activation function takes this number and changes it according to a rule. For example, it might keep it the same, cut it off if it's negative, or squeeze it between 0 and 1. This output then moves to the next neuron.

Result

The neuron outputs a transformed value instead of just passing the input directly.

Understanding that neurons don’t just pass inputs but transform them is key to grasping how neural networks learn.

2

Foundation

Why Non-Linearity is Essential

Concept: Activation functions add non-linearity, allowing networks to learn complex patterns beyond straight lines.

If neurons only added or multiplied inputs, the whole network would behave like a single straight line, no matter how many layers it had. Non-linear activation functions let the network bend and twist the data, enabling it to solve complicated problems like recognizing faces or understanding speech.

Result

Networks with non-linear activations can model complex relationships in data.

Knowing that non-linearity is what makes deep learning powerful helps you appreciate why activation functions are not optional.

3

Intermediate

Common Activation Functions Explained

🤔Before reading on: do you think all activation functions output values between 0 and 1? Commit to yes or no.

Concept: Different activation functions have different shapes and output ranges, each suited for specific tasks.

Some popular activation functions are: - Sigmoid: squeezes input between 0 and 1, like a smooth on/off switch. - ReLU (Rectified Linear Unit): outputs zero if input is negative, else passes input unchanged. - Tanh: squeezes input between -1 and 1, centered at zero. Each has strengths and weaknesses depending on the problem.

Result

You can choose an activation function that fits your network’s needs.

Recognizing the variety of activation functions helps you tailor networks for better learning and performance.

4

Intermediate

How Activation Functions Affect Learning

🤔Before reading on: do you think activation functions influence how fast a network learns? Commit to yes or no.

Concept: Activation functions impact the flow of gradients during training, affecting learning speed and stability.

During training, networks adjust weights based on errors. Activation functions shape these errors by controlling gradients. For example, ReLU helps avoid vanishing gradients, speeding up learning, while sigmoid can cause gradients to vanish, slowing training. Choosing the right activation function can make training more efficient.

Result

Networks train faster and more reliably with suitable activation functions.

Understanding the role of activation functions in training dynamics is crucial for building effective models.

5

Intermediate

Activation Functions in Output Layers

🤔Before reading on: do you think the same activation function is used in all layers? Commit to yes or no.

Concept: Output layers often use different activation functions tailored to the task, like classification or regression.

For example, in binary classification, sigmoid is used to output probabilities between 0 and 1. For multi-class classification, softmax converts outputs into probabilities that sum to 1. For regression, sometimes no activation or a linear function is used. This choice affects how the network’s output is interpreted.

Result

The network’s final output matches the problem’s requirements.

Knowing to customize output activations ensures your model’s predictions make sense for the task.

6

Advanced

Problems Like Vanishing and Exploding Gradients

🤔Before reading on: do you think all activation functions avoid training problems equally? Commit to yes or no.

Concept: Some activation functions cause gradients to become too small or too large, making training unstable.

Sigmoid and tanh can squash gradients to near zero for large inputs, causing vanishing gradients that slow or stop learning. ReLU avoids this but can cause 'dead neurons' if inputs are always negative. Advanced functions like Leaky ReLU or ELU fix these issues by allowing small gradients for negative inputs.

Result

Choosing the right activation function prevents training failures and improves model robustness.

Understanding these problems helps you pick or design activation functions that keep training healthy.

7

Expert

Custom and Learnable Activation Functions

🤔Before reading on: do you think activation functions can be learned by the network itself? Commit to yes or no.

Concept: Beyond fixed formulas, activation functions can be designed to adapt during training for better performance.

Researchers have created activation functions with parameters that the network learns, like PReLU (Parametric ReLU). These functions adjust their shape based on data, potentially improving accuracy. Designing or tuning activation functions is an advanced technique used in cutting-edge models.

Result

Networks can self-optimize their activation behavior for specific tasks.

Knowing that activation functions can be flexible and trainable opens doors to more powerful and adaptive models.

Under the Hood

Activation functions apply a mathematical operation to the weighted sum of inputs plus bias inside a neuron. This operation transforms the input signal into an output signal that is passed to the next layer. During training, the derivative of the activation function is used to calculate gradients for backpropagation, guiding weight updates. The shape and properties of the activation function directly influence gradient flow and network learning dynamics.

Why designed this way?

Activation functions were designed to introduce non-linearity because linear models cannot capture complex patterns. Early functions like sigmoid were inspired by biological neurons' firing rates. Over time, functions like ReLU were introduced to solve training issues like vanishing gradients and to speed up computation. The design balances mathematical properties, biological inspiration, and practical training needs.

Input Vector ──▶ Weighted Sum (Σ weights * inputs + bias) ──▶ Activation Function ──▶ Output

Backpropagation:
Output Error ──▶ Derivative of Activation ──▶ Gradient Calculation ──▶ Weight Update

Myth Busters - 4 Common Misconceptions

Quick: Do you think sigmoid activation always helps training deep networks? Commit to yes or no.

Common Belief: Sigmoid activation is always a good choice because it outputs probabilities between 0 and 1.

Reality: Sigmoid often causes vanishing gradients in deep networks, making training slow or stuck.

Why it matters: Using sigmoid blindly can prevent deep networks from learning effectively, wasting time and resources.

Quick: Do you think ReLU activation never causes problems? Commit to yes or no.

Common Belief: ReLU is perfect and never causes issues during training.

Reality: ReLU can cause 'dead neurons' that stop learning if inputs are always negative.

Why it matters: Ignoring this can lead to parts of the network becoming useless, reducing model capacity.

Quick: Do you think all layers in a network should use the same activation function? Commit to yes or no.

Common Belief: Using the same activation function everywhere is best for consistency.

Reality: Different layers, especially output layers, often need different activation functions suited to their role.

Why it matters: Wrong output activations can produce meaningless predictions, hurting model usefulness.

Quick: Do you think activation functions only affect output values, not training? Commit to yes or no.

Common Belief: Activation functions just change output values; they don’t affect how the network learns.

Reality: Activation functions strongly influence gradient flow and thus the speed and success of training.

Why it matters: Ignoring this leads to poor training choices and suboptimal models.

Expert Zone

1

Some activation functions work better with specific weight initialization methods to maintain stable gradients.

2

Learnable activation functions can adapt to data but add complexity and risk overfitting if not regularized.

3

Activation functions can interact with normalization layers, affecting overall network behavior in subtle ways.

When NOT to use

Avoid using sigmoid or tanh in very deep networks due to vanishing gradients; prefer ReLU or its variants. For output layers, do not use ReLU when probabilities are needed; use sigmoid or softmax instead. When interpretability is critical, linear activations might be preferred. Alternatives include maxout units or attention mechanisms depending on the architecture.

Production Patterns

In production, ReLU and its variants dominate hidden layers for efficiency and performance. Output layers use sigmoid for binary classification and softmax for multi-class. Custom activations are rare but appear in specialized models like GANs or reinforcement learning. Monitoring neuron activation distributions helps detect dead neurons or saturation.

Connections

Biological Neurons

Activation functions mimic the firing behavior of biological neurons.

Understanding biological neurons helps appreciate why activation functions squash or threshold signals, grounding artificial networks in natural processes.

Signal Processing

Activation functions act like filters or signal transformers in signal processing.

Knowing signal processing concepts clarifies how activation functions shape and control information flow in networks.

Nonlinear Dynamics

Activation functions introduce nonlinearity, a key concept in nonlinear dynamic systems.

Recognizing activation functions as nonlinear operators connects neural networks to broader mathematical systems that exhibit complex behavior.

Common Pitfalls

#1 Using sigmoid activation in all layers of a deep network.

Wrong approach:

model.add(Dense(64, activation='sigmoid'))
model.add(Dense(64, activation='sigmoid'))
model.add(Dense(10, activation='softmax'))

Correct approach:

model.add(Dense(64, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(10, activation='softmax'))

Root cause:

Misunderstanding that sigmoid causes vanishing gradients in deep layers, slowing or stopping learning.

#2 Using ReLU activation in output layer for classification.

Wrong approach:

model.add(Dense(10, activation='relu'))

Correct approach:

model.add(Dense(10, activation='softmax'))

Root cause:

Confusing hidden layer activations with output layer requirements; output needs probability distribution.

#3 Ignoring dead neurons caused by ReLU.

Wrong approach:

model.add(Dense(64, activation='relu')) # no monitoring or fix

Correct approach:

model.add(Dense(64, activation='leaky_relu')) # allows small gradient for negative inputs

Root cause:

Not realizing ReLU outputs zero for negative inputs, causing some neurons to stop updating.

Key Takeaways

Activation functions add essential non-linearity that enables neural networks to learn complex patterns.

Choosing the right activation function affects both the network’s ability to learn and the quality of its predictions.

Common functions like ReLU, sigmoid, and tanh each have strengths and weaknesses that impact training dynamics.

Output layers require activation functions suited to the task, such as softmax for multi-class classification.

Advanced techniques include learnable activation functions that adapt during training for improved performance.