Why deployment delivers value in ML Python - Why Metrics Matter

Metrics & Evaluation - Why deployment delivers value

Which metric matters for this concept and WHY

When we talk about deployment delivering value, the key metrics are real-world performance metrics like accuracy, precision, recall, and latency measured on live data. These metrics matter because they show how well the model helps solve the actual problem after deployment, not just during training.

For example, a model with high accuracy in the lab but slow response time or poor precision on live data won't deliver value. So, we focus on metrics that reflect real user impact and operational efficiency.

Confusion matrix or equivalent visualization (ASCII)

      Confusion Matrix on Live Data:

          Predicted Positive   Predicted Negative
    Actual Positive      85 (TP)             15 (FN)
    Actual Negative      10 (FP)             90 (TN)

    Total samples = 200
    

This matrix helps us calculate precision, recall, and accuracy on real data after deployment.

Precision vs Recall tradeoff with concrete examples

After deployment, we often face a tradeoff between precision and recall depending on the use case:

• High precision needed: Email spam filter should avoid marking good emails as spam. So, precision must be high to reduce false alarms.
• High recall needed: Disease detection system should catch as many sick patients as possible. So, recall must be high to avoid missing cases.

Deployment lets us monitor these tradeoffs live and adjust thresholds or models to maximize value.

What "good" vs "bad" metric values look like for this use case

Good deployment metrics:

• Accuracy above 85% on live data
• Precision and recall balanced according to business needs (e.g., both above 80%)
• Low latency (fast predictions under 100ms)
• Stable metrics over time (no sudden drops)

Bad deployment metrics:

• High accuracy in training but below 70% live accuracy
• Very low recall (e.g., 20%) missing many real cases
• High false positives causing user frustration
• Slow response times making the system unusable

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. For example, 95% accuracy by always predicting the majority class.
• Data leakage: If deployment data leaks training info, metrics look great but fail in real use.
• Overfitting: Model performs well on training but poorly on live data, showing deployment metrics drop.
• Ignoring latency: A model with good accuracy but slow predictions reduces user satisfaction.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. The 98% accuracy is misleading because fraud cases are rare. The very low recall (12%) means the model misses most fraud cases, which is dangerous. In fraud detection, catching fraud (high recall) is more important than overall accuracy.

Key Result

Deployment value depends on real-world metrics like balanced precision and recall, latency, and stable performance on live data.