ONNX Runtime inference in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - ONNX Runtime inference

Which metric matters for ONNX Runtime inference and WHY

When using ONNX Runtime for inference, the main goal is to get fast and accurate predictions from your model. The key metrics to check are inference latency (how fast the model predicts) and prediction accuracy (how correct the predictions are). Latency matters because ONNX Runtime is often used to speed up models in real-time apps. Accuracy matters because a faster model is useless if it gives wrong answers.

Confusion matrix example for ONNX Runtime inference

Suppose your ONNX model predicts if an email is spam or not. After running inference on 100 emails, you get:

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positives (TP) = 40          | False Negatives (FN) = 10       |
      | False Positives (FP) = 5          | True Negatives (TN) = 45        |
    

This matrix helps calculate precision, recall, and accuracy to check if ONNX Runtime inference keeps prediction quality.

Precision vs Recall tradeoff with ONNX Runtime inference

Imagine your ONNX model detects fraud in transactions. If you want to avoid missing fraud cases, focus on high recall. But if you want to avoid false alarms, focus on high precision. ONNX Runtime helps by making inference fast, so you can try different thresholds quickly to find the best balance.

For example, increasing recall might catch more fraud but also flag more good transactions wrongly. ONNX Runtime lets you test these tradeoffs efficiently.

What good vs bad metric values look like for ONNX Runtime inference

Good: Low latency (e.g., under 10 milliseconds per prediction), accuracy close to the original model (e.g., 95%+), and balanced precision and recall depending on use case.

Bad: High latency (e.g., over 100 milliseconds), accuracy drop (e.g., below 80%), or very low precision or recall indicating poor prediction quality after conversion to ONNX.

Common pitfalls in ONNX Runtime inference metrics

• Accuracy drop: Sometimes converting to ONNX changes model behavior, causing lower accuracy.
• Ignoring latency: Focusing only on accuracy and ignoring inference speed misses the main benefit of ONNX Runtime.
• Data leakage: Testing on data used in training inflates accuracy falsely.
• Overfitting signs: Very high accuracy on test but poor real-world results means model or data issues, not ONNX Runtime.

Self-check question

Your ONNX Runtime inference model has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the very low recall means the model misses most fraud cases. For fraud detection, recall is critical because missing fraud is costly. You should improve recall before using this model in production.

Key Result

ONNX Runtime inference success depends on maintaining original model accuracy while significantly reducing prediction latency.