Activation functions (ReLU, sigmoid, softmax) in TensorFlow - Model Pipeline Trace

Model Pipeline - Activation functions (ReLU, sigmoid, softmax)

This pipeline shows how data moves through a simple neural network using three common activation functions: ReLU, sigmoid, and softmax. These functions help the model learn by adding non-linear behavior and producing probabilities for classification.

Data Flow - 4 Stages

1Input Layer

1 row x 4 columns→Raw input features representing 4 numeric values→1 row x 4 columns

[2.0, -1.0, 0.5, 3.0]

↓

2Hidden Layer with ReLU

1 row x 4 columns→Linear transformation (weights and bias) followed by ReLU activation (max(0, x))→1 row x 3 columns

Input [2.0, -1.0, 0.5, 3.0] -> Linear output [1.5, -0.5, 2.0] -> ReLU output [1.5, 0.0, 2.0]

↓

3Hidden Layer with Sigmoid

1 row x 3 columns→Linear transformation followed by sigmoid activation (output between 0 and 1)→1 row x 3 columns

Input [1.5, 0.0, 2.0] -> Linear output [0.8, -1.2, 0.5] -> Sigmoid output [0.69, 0.23, 0.62]

↓

4Output Layer with Softmax

1 row x 3 columns→Linear transformation followed by softmax activation (outputs sum to 1, representing class probabilities)→1 row x 3 columns

Input [0.69, 0.23, 0.62] -> Linear output [2.0, 1.0, 0.1] -> Softmax output [0.66, 0.24, 0.10]

Training Trace - Epoch by Epoch

Loss
1.2 |*       
1.0 | *      
0.8 |  *     
0.6 |   *    
0.4 |    *   
    +---------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.45	Loss starts high, accuracy low as model begins learning
2	0.9	0.60	Loss decreases, accuracy improves as activations help model learn
3	0.7	0.72	Model continues to improve with clearer decision boundaries
4	0.5	0.80	Loss decreases steadily, accuracy rises showing good learning
5	0.4	0.85	Model converges with lower loss and higher accuracy

Prediction Trace - 4 Layers

Layer 1: Input Layer

Layer 2: Hidden Layer with ReLU

Layer 3: Hidden Layer with Sigmoid

Layer 4: Output Layer with Softmax

Model Quiz - 3 Questions

What does the ReLU activation function do to negative input values?

A Converts them to probabilities

B Sets them to zero

C Leaves them unchanged

D Maps them between 0 and 1

Key Insight

Activation functions like ReLU, sigmoid, and softmax add important non-linear transformations that help neural networks learn complex patterns and produce meaningful outputs such as probabilities for classification.