Retraining strategies in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Retraining strategies

Which metric matters for Retraining strategies and WHY

When we retrain a model, we want to see if it learns better from new data. Key metrics to watch are validation loss and validation accuracy. These tell us if the model is improving on unseen data, not just memorizing old data.

Also, watch precision and recall if your task is classification. They show if the model is correctly finding important cases (recall) and avoiding false alarms (precision).

Finally, track training time and resource use to balance model quality with cost.

Confusion matrix example during retraining

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |

      Example after retraining:
      TP = 85, FP = 15, TN = 900, FN = 20

      Total samples = 85 + 15 + 900 + 20 = 1020

      Precision = TP / (TP + FP) = 85 / (85 + 15) = 0.85
      Recall = TP / (TP + FN) = 85 / (85 + 20) = 0.81
    

Precision vs Recall tradeoff in retraining

When retraining, improving one metric can lower the other. For example:

• High precision means fewer false alarms. Good if false alarms are costly, like spam filters.
• High recall means fewer missed cases. Important if missing a case is dangerous, like in medical diagnosis.

Retraining can help find a better balance by learning from new examples. But watch if improving recall drops precision too much or vice versa.

Good vs Bad metric values for retraining

Good retraining results:

• Validation accuracy improves or stays stable.
• Precision and recall both increase or maintain a good balance.
• Loss on validation data decreases.
• No big jump in training time or resource use.

Bad retraining results:

• Validation accuracy drops or fluctuates wildly.
• Precision or recall drops significantly.
• Validation loss increases, showing worse generalization.
• Training time grows too long without metric gains.

Common pitfalls in retraining metrics

• Overfitting: Training loss goes down but validation loss goes up. Model memorizes new data but fails on unseen data.
• Data leakage: New training data accidentally includes test data, inflating metrics falsely.
• Ignoring metric drift: Not checking if metrics degrade over time due to changing data patterns.
• Using only accuracy: Accuracy can be misleading if classes are imbalanced. Use precision and recall too.

Self-check question

Your model after retraining has 98% accuracy but only 12% recall on fraud cases. Is it good for production?

Answer: No. Even though accuracy is high, the model misses 88% of fraud cases (low recall). This means many frauds go undetected, which is risky. You should improve recall before using it in production.

Key Result

Retraining success is shown by improved validation metrics like loss, accuracy, precision, and recall, balanced with training cost.