Activation functions in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Activation functions

Which metric matters for Activation Functions and WHY

Activation functions affect how well a model learns and predicts. The key metrics to watch are loss and accuracy. Loss shows how far off predictions are during training. Accuracy shows how often the model predicts correctly. Good activation functions help lower loss and raise accuracy by allowing the model to learn complex patterns.

Confusion Matrix Example

For a classification model using activation functions like ReLU or Sigmoid, the confusion matrix helps us see prediction results:

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    85    |   15
      Negative           |    10    |   90
    

Here, TP=85, FP=10, FN=15, TN=90. Activation functions influence these numbers by shaping model outputs.

Precision vs Recall Tradeoff

Activation functions impact model sensitivity. For example, Sigmoid outputs probabilities, helping balance precision and recall. ReLU can speed up learning but may cause some neurons to "die" (stop learning), affecting recall.

In a spam filter, high precision means fewer good emails marked as spam. Using activation functions that help the model focus on clear signals improves precision.

In medical diagnosis, high recall means catching most sick patients. Activation functions that allow subtle signals to pass help improve recall.

Good vs Bad Metric Values for Activation Functions

Good: Low loss (e.g., below 0.2), high accuracy (above 85%), balanced precision and recall (both above 80%). This means the activation function helps the model learn well.

Bad: High loss (above 1.0), low accuracy (below 60%), very low recall or precision (below 50%). This suggests the activation function may cause learning problems like vanishing gradients or dead neurons.

Common Pitfalls with Activation Functions

• Vanishing gradients: Functions like Sigmoid can squash values, making learning slow or stuck.
• Dead neurons: ReLU can output zero for many inputs, causing some neurons to stop updating.
• Choosing wrong function: Using linear activation in hidden layers limits model power.
• Ignoring data scale: Activation functions may behave poorly if input data is not normalized.

Self Check

Your model uses ReLU activation and shows 98% accuracy but training loss stays high and validation accuracy is low. Is this good?

Answer: No. High accuracy with high loss and low validation accuracy suggests overfitting or dead neurons. The activation function might cause some neurons to stop learning. Try Leaky ReLU or check data preprocessing.

Key Result

Activation functions influence loss and accuracy by enabling models to learn complex patterns effectively.