ONNX Runtime in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - ONNX Runtime

Which metric matters for ONNX Runtime and WHY

ONNX Runtime is a tool to run machine learning models fast and efficiently. When using it for computer vision, the key metrics are inference speed and model accuracy. Speed matters because ONNX Runtime helps models make predictions quickly, which is important for real-time tasks like object detection in videos. Accuracy matters because a fast model that makes wrong predictions is not useful. So, we want to measure how fast the model runs and how correct its predictions are.

Confusion matrix example for ONNX Runtime model

    Confusion Matrix (Example for a 2-class image classifier):

          Predicted
          Cat   Dog
    Actual
    Cat   85    15
    Dog   10    90

    Total samples = 85 + 15 + 10 + 90 = 200

    True Positives (TP) = 85 (correctly predicted Cat)
    False Positives (FP) = 10 (Dog predicted as Cat)
    True Negatives (TN) = 90 (correctly predicted Dog)
    False Negatives (FN) = 15 (Cat predicted as Dog)
    

Precision vs Recall tradeoff with ONNX Runtime

Imagine ONNX Runtime runs a model to detect cats in photos.

• Precision means: Of all photos predicted as cats, how many really are cats? High precision means fewer false alarms.
• Recall means: Of all actual cat photos, how many did the model find? High recall means fewer missed cats.

If ONNX Runtime speeds up the model but the model misses many cats (low recall), it is not good for applications like pet monitoring. If it finds many cats but also mistakes dogs for cats (low precision), it causes false alerts.

So, ONNX Runtime helps balance speed with maintaining good precision and recall.

Good vs Bad metric values for ONNX Runtime models

Good values:

• Accuracy above 90% on test images
• Precision and recall both above 85%
• Inference time reduced by 50% compared to original model

Bad values:

• Accuracy below 70%, meaning many wrong predictions
• Precision or recall below 50%, causing many false alarms or misses
• Inference speed not improved or slower, defeating ONNX Runtime's purpose

Common pitfalls when evaluating ONNX Runtime models

• Ignoring accuracy drop: Speeding up with ONNX Runtime may reduce accuracy if model conversion is not done carefully.
• Data leakage: Testing on data the model saw during training gives false high accuracy.
• Overfitting: Model performs well on training but poorly on new images, misleading metrics.
• Measuring only speed: Fast inference is good but useless if predictions are wrong.

Self-check question

Your ONNX Runtime model runs 3 times faster than the original but has 98% accuracy and only 12% recall on detecting a rare object. Is it good for production? Why or why not?

Answer: No, it is not good. Although the model is fast and has high overall accuracy, the very low recall means it misses most rare objects. For rare object detection, missing them is critical, so recall must be higher even if speed is slightly lower.

Key Result

ONNX Runtime models must balance fast inference speed with high accuracy, precision, and recall to be effective.