Gradient descent optimization in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Gradient descent optimization

Which metric matters for Gradient Descent Optimization and WHY

Gradient descent is about finding the lowest point on a hill, which means minimizing a loss function. The key metric here is the loss value itself, such as Mean Squared Error (MSE) for regression or Cross-Entropy Loss for classification.

We want the loss to get smaller with each step, showing the model is learning. Tracking loss reduction over time tells us if gradient descent is working well.

Sometimes, we also look at training accuracy or validation accuracy to see if the model is improving in real terms, but loss is the main guide for gradient descent.

Confusion Matrix or Equivalent Visualization

Gradient descent itself does not produce a confusion matrix because it optimizes loss, not classification directly. However, after training, we can evaluate the model with a confusion matrix.

Confusion Matrix Example (Binary Classification):

          Predicted
          0     1
Actual 0  TN    FP
       1  FN    TP

Where:
- TP: True Positives
- FP: False Positives
- TN: True Negatives
- FN: False Negatives

Loss guides gradient descent to reduce errors that lead to these counts.
    

Precision vs Recall Tradeoff with Gradient Descent

Gradient descent minimizes loss but does not directly control precision or recall. However, the choice of loss function and how well gradient descent optimizes it affects these metrics.

For example, if the loss function penalizes false negatives heavily, gradient descent will try to reduce them, improving recall.

In real life, imagine a spam filter: if missing spam is worse, the loss can be designed to focus on recall. Gradient descent then tries to find model settings that improve recall, even if precision drops a bit.

What Good vs Bad Metric Values Look Like for Gradient Descent

Good: Loss steadily decreases over training steps, showing the model is learning. For example, MSE dropping from 1.0 to 0.1 means better predictions.

Bad: Loss stays flat or bounces around, meaning gradient descent is stuck or not working. This could be due to a too-large step size or poor data.

Also, if training loss decreases but validation loss increases, it means overfitting, which is a warning sign.

Common Pitfalls in Gradient Descent Metrics

• Accuracy Paradox: High accuracy can be misleading if classes are imbalanced. Gradient descent focuses on loss, which can help avoid this.
• Data Leakage: If validation data leaks into training, loss looks better but model won't generalize.
• Overfitting: Training loss goes down but validation loss goes up. Gradient descent is fitting noise, not real patterns.
• Learning Rate Issues: Too high learning rate causes loss to jump or diverge; too low makes training slow.

Self Check

Your model's training loss is decreasing steadily, but validation loss is increasing. Is gradient descent working well? Why or why not?

Answer: Gradient descent is minimizing training loss, but the model is overfitting. It learns training data too well but fails to generalize. This means gradient descent alone is not enough; you need regularization or more data.

Key Result

Loss reduction over training steps is the key metric to evaluate gradient descent optimization effectiveness.