FastAPI for model serving in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - FastAPI for model serving

Which metric matters for FastAPI model serving and WHY

When serving a machine learning model with FastAPI, key metrics to watch are latency and throughput. Latency measures how fast the model responds to a request, important for user experience. Throughput measures how many requests the server can handle per second, important for scaling. Also, monitor prediction accuracy to ensure the model gives correct results. These metrics help balance speed and correctness in real-time use.

Confusion matrix example for model predictions

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20    
      Negative           |    10    |   90    

      Total samples = 80 + 20 + 10 + 90 = 200
    

From this matrix, we calculate:

• Precision = 80 / (80 + 10) = 0.89
• Recall = 80 / (80 + 20) = 0.80
• F1 Score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84

These metrics show how well the model performs when served via FastAPI.

Precision vs Recall tradeoff in model serving

In FastAPI model serving, sometimes you must choose between precision and recall:

• High Precision: The model rarely gives wrong positive predictions. Useful when false alarms are costly, like spam filters.
• High Recall: The model catches most positive cases. Important when missing a positive is dangerous, like disease detection.

Depending on your app, you might tune the model or threshold to favor one metric. FastAPI lets you quickly update and test these changes.

Good vs Bad metric values for FastAPI model serving

Good metrics:

• Latency under 100 ms for fast user response
• Throughput of hundreds of requests per second for scalability
• Precision and recall above 0.8 for reliable predictions

Bad metrics:

• Latency over 1 second causing slow responses
• Throughput too low to handle user load
• Precision or recall below 0.5 indicating poor prediction quality

Common pitfalls in FastAPI model serving metrics

• Ignoring latency spikes: Occasional slow responses hurt user experience.
• Overfitting: Model shows great accuracy on test data but fails in real-time serving.
• Data leakage: Training data leaks into test data, inflating accuracy falsely.
• Not monitoring throughput: Server crashes under load if throughput is not tested.
• Confusing accuracy with precision or recall: Accuracy alone can be misleading if classes are imbalanced.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. Even though accuracy is high, the recall is very low. This means the model misses 88% of fraud cases, which is dangerous. In fraud detection, high recall is critical to catch as many frauds as possible. You should improve recall before deploying.

Key Result

FastAPI model serving requires balancing low latency and high throughput with good precision and recall for reliable predictions.