Experiment tracking (MLflow) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Experiment tracking (MLflow)

Which metric matters for Experiment Tracking and WHY

Experiment tracking helps you keep a clear record of your model tests. The key metrics to track depend on your goal, like accuracy, loss, precision, or recall. Tracking these lets you compare models easily and pick the best one. Without tracking, you might forget which model worked best or why.

Confusion Matrix or Equivalent Visualization

While MLflow itself does not create confusion matrices, it stores metrics like True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). You can log these values and then visualize the confusion matrix externally to understand model performance.

Confusion Matrix Example:

          Predicted
          P     N
Actual P  TP=50 FN=10
       N  FP=5  TN=35

Total samples = 50 + 10 + 5 + 35 = 100
    

Precision vs Recall Tradeoff with Concrete Examples

When tracking experiments, logging both precision and recall helps you see tradeoffs. For example:

• Spam filter: High precision means fewer good emails marked as spam. You want to avoid false alarms.
• Cancer detection: High recall means catching most cancer cases, even if some false alarms happen.

MLflow lets you track these metrics side by side to choose the best balance for your needs.

What "Good" vs "Bad" Metric Values Look Like for Experiment Tracking

Good experiment tracking means:

• Consistent logging of all important metrics (accuracy, loss, precision, recall, F1).
• Clear naming so you know which model and parameters produced which results.
• Comparing metrics across runs to find improvements.

Bad tracking means missing metrics, unclear records, or no way to compare runs. This leads to confusion and wasted time.

Common Pitfalls in Experiment Tracking Metrics

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. Tracking precision and recall helps avoid this.
• Data leakage: If your experiment accidentally uses future data, metrics look too good but won't work in real life.
• Overfitting indicators: Tracking training vs validation metrics helps spot if your model only memorizes training data.
• Inconsistent metric definitions: Make sure you log metrics the same way every time.

Self Check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, this model is not good for fraud detection. Even though accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. For fraud, catching as many frauds as possible (high recall) is critical, even if some false alarms happen.

Key Result

Experiment tracking focuses on logging key metrics like accuracy, precision, and recall to compare models and avoid pitfalls like misleading accuracy or data leakage.