Activation functions (ReLU, sigmoid, softmax) in TensorFlow

Introduction

Activation functions help a neural network learn complex patterns by deciding which signals to pass forward. They add non-linearity so the model can solve real-world problems.

When building a neural network to classify images or text.

When you want the model to decide if a neuron should be active or not.

When you need outputs as probabilities for classification tasks.

When you want to avoid negative outputs in your model.

When you want to squash outputs between 0 and 1 for binary decisions.

Syntax

TensorFlow

tf.keras.layers.Activation('relu')
tf.keras.layers.Activation('sigmoid')
tf.keras.layers.Activation('softmax')

ReLU outputs zero for negative inputs and the input itself if positive.

Sigmoid squashes input values between 0 and 1, useful for binary outputs.

Examples

This creates a layer with 10 neurons using ReLU activation.

TensorFlow

layer = tf.keras.layers.Dense(10, activation='relu')

This creates a single neuron layer with sigmoid activation for binary output.

TensorFlow

layer = tf.keras.layers.Dense(1, activation='sigmoid')

This creates a layer with 3 neurons using softmax to output probabilities for 3 classes.

TensorFlow

layer = tf.keras.layers.Dense(3, activation='softmax')

Sample Model

This code shows how ReLU, sigmoid, and softmax activation functions transform the same input array. It prints the input and the outputs after applying each activation.

TensorFlow

import tensorflow as tf
import numpy as np

# Sample input data
x = np.array([[-1.0, 0.0, 1.0, 2.0]])

# Define layers with different activations
relu_layer = tf.keras.layers.Activation('relu')
sigmoid_layer = tf.keras.layers.Activation('sigmoid')
softmax_layer = tf.keras.layers.Activation('softmax')

# Apply activations
relu_output = relu_layer(x)
sigmoid_output = sigmoid_layer(x)
softmax_output = softmax_layer(x)

print('Input:', x)
print('ReLU output:', relu_output.numpy())
print('Sigmoid output:', sigmoid_output.numpy())
print('Softmax output:', softmax_output.numpy())

Important Notes

ReLU is simple and fast but can cause some neurons to 'die' if they always output zero.

Sigmoid outputs can saturate near 0 or 1, which slows learning for deep networks.

Softmax is used only in the output layer for multi-class classification to get probabilities.

Summary

Activation functions add non-linearity to neural networks.

ReLU is good for hidden layers to keep positive signals.

Sigmoid and softmax are used for output layers to get probabilities.