Activation functions in ML Python - Model Pipeline Trace

Model Pipeline - Activation functions

This pipeline shows how activation functions help a neural network learn by adding non-linear behavior. They transform the data after each layer so the model can understand complex patterns.

Data Flow - 5 Stages

1Input data

1 sample x 3 features→Raw input features fed into the model→1 sample x 3 features

[0.5, -1.2, 3.3]

↓

2Linear transformation

1 sample x 3 features→Multiply inputs by weights and add bias→1 sample x 2 neurons

[1.1, -0.7]

↓

3Activation function (ReLU)

1 sample x 2 neurons→Apply ReLU to keep positive values, zero out negatives→1 sample x 2 neurons

[1.1, 0.0]

↓

4Linear transformation

1 sample x 2 neurons→Multiply by next layer weights and add bias→1 sample x 2 neurons

[0.3, -0.4]

↓

5Activation function (Sigmoid)

1 sample x 2 neurons→Apply sigmoid to squash values between 0 and 1→1 sample x 2 neurons

[0.574, 0.401]

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |*   
0.3 |*   
    +-----
    1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Loss starts high, accuracy is low as model begins learning
2	0.50	0.72	Loss decreases, accuracy improves as activations help model learn
3	0.40	0.80	Model learns better patterns, activation functions enable non-linear learning
4	0.35	0.85	Loss continues to drop, accuracy rises steadily
5	0.30	0.88	Training converges, activation functions help model capture complex data

Prediction Trace - 5 Layers

Layer 1: Input layer

Layer 2: First linear layer

Layer 3: ReLU activation

Layer 4: Second linear layer

Layer 5: Sigmoid activation

Model Quiz - 3 Questions

What does the ReLU activation function do to negative input values?

A Changes them to zero

B Keeps them the same

C Changes them to one

D Inverts their sign

Key Insight

Activation functions add important non-linear transformations that let neural networks learn complex patterns beyond simple straight lines. Without them, models would be limited to only simple relationships.