TorchScript for production in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - TorchScript for production

Which metric matters for TorchScript in production and WHY

TorchScript is used to make PyTorch models ready for production. The key metric here is inference latency, which means how fast the model can give answers. Faster models make apps feel smooth and save money on servers.

Another important metric is model accuracy. When converting to TorchScript, the model should keep its original accuracy. If accuracy drops, the model is not reliable.

Finally, model size matters because smaller models use less memory and load faster.

Confusion matrix or equivalent visualization

For TorchScript, we check if predictions before and after scripting match. Here is a simple example with 5 samples:

Original Model Predictions: [1, 0, 1, 1, 0]
TorchScript Predictions:     [1, 0, 1, 1, 0]
    

All predictions match, so the conversion kept accuracy.

Precision vs Recall tradeoff with concrete examples

While TorchScript itself does not change precision or recall, it is important to verify these metrics stay the same after conversion.

For example, if your model detects spam emails:

• Precision means how many emails marked as spam really are spam.
• Recall means how many spam emails your model finds out of all spam emails.

If TorchScript conversion lowers recall, your app might miss spam emails. If it lowers precision, good emails might be wrongly marked as spam.

So, always check precision and recall before and after scripting to keep the balance.

What "good" vs "bad" metric values look like for TorchScript

Good:

• Inference latency reduced by 20% or more compared to original model.
• Accuracy, precision, recall unchanged or within 1% difference.
• Model size reduced or stays the same.

Bad:

• Inference latency is the same or slower.
• Accuracy drops by more than 2%.
• Model size grows significantly, causing memory issues.

Metrics pitfalls when using TorchScript

• Accuracy drop: Sometimes scripting changes model behavior, causing lower accuracy.
• Data leakage: If test data leaks into training, metrics look better but are not real.
• Overfitting: Good training metrics but poor real-world performance.
• Ignoring latency: Only checking accuracy but ignoring speed can cause slow apps.
• Not testing on real inputs: TorchScript may fail on some input types if not tested well.

Self-check question

Your TorchScript model has 98% accuracy but 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. For fraud detection, high recall is critical to catch as many frauds as possible.

Key Result

TorchScript should keep model accuracy stable while reducing inference latency for smooth production use.