FastAPI for model serving in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - FastAPI for model serving

Which metric matters for FastAPI model serving and WHY

When serving a machine learning model with FastAPI, key metrics to watch are latency and throughput. Latency measures how fast the model responds to a request, important for user experience. Throughput measures how many requests the server can handle per second, important for scaling. Also, monitor prediction accuracy to ensure the model gives correct results. These metrics help balance speed and correctness in real-time use.

Confusion matrix example for model predictions

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20    
      Negative           |    10    |   90    

      Total samples = 80 + 20 + 10 + 90 = 200
    

From this matrix, we calculate:

• Precision = 80 / (80 + 10) = 0.89
• Recall = 80 / (80 + 20) = 0.80
• F1 Score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84

These metrics show how well the model performs when served via FastAPI.

Precision vs Recall tradeoff in model serving

In FastAPI model serving, sometimes you must choose between precision and recall:

• High Precision: The model rarely gives wrong positive predictions. Useful when false alarms are costly, like spam filters.
• High Recall: The model catches most positive cases. Important when missing a positive is dangerous, like disease detection.

Depending on your app, you might tune the model or threshold to favor one metric. FastAPI lets you quickly update and test these changes.

Good vs Bad metric values for FastAPI model serving

Good metrics:

• Latency under 100 ms for fast user response
• Throughput of hundreds of requests per second for scalability
• Precision and recall above 0.8 for reliable predictions

Bad metrics:

• Latency over 1 second causing slow responses
• Throughput too low to handle user load
• Precision or recall below 0.5 indicating poor prediction quality

Common pitfalls in FastAPI model serving metrics

• Ignoring latency spikes: Occasional slow responses hurt user experience.
• Overfitting: Model shows great accuracy on test data but fails in real-time serving.
• Data leakage: Training data leaks into test data, inflating accuracy falsely.
• Not monitoring throughput: Server crashes under load if throughput is not tested.
• Confusing accuracy with precision or recall: Accuracy alone can be misleading if classes are imbalanced.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. Even though accuracy is high, the recall is very low. This means the model misses 88% of fraud cases, which is dangerous. In fraud detection, high recall is critical to catch as many frauds as possible. You should improve recall before deploying.

Key Result

FastAPI model serving requires balancing low latency and high throughput with good precision and recall for reliable predictions.

Practice

1. What is the main purpose of using FastAPI in machine learning model serving?

easy

A. To create a web service that allows users to send data and get model predictions

B. To train machine learning models faster on GPUs

C. To visualize data with interactive charts

D. To store large datasets efficiently

Solution

1. Step 1: Understand FastAPI's role

   FastAPI is a web framework used to build APIs that can serve machine learning models.

2. Step 2: Identify the main use in ML serving

   It allows sending input data and receiving predictions via HTTP requests, making models accessible.

3. Final Answer:

   To create a web service that allows users to send data and get model predictions -> Option A

4. Quick Check:

   FastAPI serves ML models as web APIs = D [OK]

Hint: FastAPI is for serving models, not training or visualization [OK]

Common Mistakes:

• Confusing FastAPI with training libraries
• Thinking FastAPI is for data storage
• Assuming FastAPI is for data visualization

2. Which of the following is the correct way to define a POST endpoint in FastAPI for model prediction?

easy

A. app.post('/predict')

B. app.put('/predict')

C. app.get('/predict')

D. app.delete('/predict')

Solution

1. Step 1: Recall HTTP methods for sending data

   POST is used to send data to the server, suitable for prediction requests.

2. Step 2: Match method to FastAPI syntax

   FastAPI uses @app.post decorator to define POST endpoints.

3. Final Answer:

   app.post('/predict') -> Option A

4. Quick Check:

   POST endpoint for sending data = app.post [OK]

Hint: Use POST to send data for predictions in FastAPI [OK]

Common Mistakes:

• Using GET instead of POST for sending input data
• Confusing HTTP methods like PUT or DELETE
• Missing the @ decorator syntax

3. Given the following FastAPI code snippet, what will be the output when sending JSON {"text": "hello"} to /predict?

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class InputData(BaseModel):
    text: str

@app.post('/predict')
def predict(data: InputData):
    return {"length": len(data.text)}

medium

A. {"text": 5}

B. {"length": "hello"}

C. {"length": 5}

D. Error: Missing return type

Solution

1. Step 1: Understand input and output

   The input JSON has "text": "hello". The function returns the length of this string.

2. Step 2: Calculate length of 'hello'

   "hello" has 5 characters, so the output JSON is {"length": 5}.

3. Final Answer:

   {"length": 5} -> Option C

4. Quick Check:

   len('hello') = 5 [OK]

Hint: Count characters in input string for length output [OK]

Common Mistakes:

• Returning the string instead of its length
• Confusing keys in the output JSON
• Expecting a syntax or runtime error

4. Identify the error in this FastAPI model serving code:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class InputData(BaseModel):
    value: int

@app.post('/predict')
def predict(data: InputData):
    return {"result": data.value * 2}

# Client sends JSON: {"value": "10"}

medium

A. Missing return statement

B. Type error because 'value' should be int but client sent string

C. Syntax error in function definition

D. No error, code works fine

Solution

1. Step 1: Check data type in Pydantic model

   InputData expects 'value' as int, client sends string "10".

2. Step 2: Understand Pydantic validation and coercion

   Pydantic automatically converts compatible strings like "10" to int 10, so no validation error occurs.

3. Final Answer:

   No error, code works fine -> Option D

4. Quick Check:

   Pydantic coerces "10" to 10 [OK]

Hint: Pydantic automatically converts compatible types like str to int [OK]

Common Mistakes:

• Thinking Pydantic doesn't auto-convert string to int
• Expecting a type validation error
• Overlooking Pydantic's coercion feature

5. You want to serve a sentiment analysis model using FastAPI. The model expects a list of texts and returns a list of sentiment scores. Which is the best way to define the input and output in FastAPI?

hard

A. Accept raw JSON string and return a plain string

B. Use a Pydantic model with a field 'texts' as List[str] and return List[float]

C. Use query parameters for each text input and return a single float

D. Use a Pydantic model with a single string field and return a dictionary with one score

Solution

1. Step 1: Define input for multiple texts

   Since the model expects multiple texts, input should be a list of strings, best defined with Pydantic List[str].

2. Step 2: Define output for multiple scores

   The model returns multiple sentiment scores, so output should be a list of floats matching input size.

3. Final Answer:

   Use a Pydantic model with a field 'texts' as List[str] and return List[float] -> Option B

4. Quick Check:

   Input/output types match model expectations [OK]

Hint: Match input/output types to model's batch processing [OK]

Common Mistakes:

• Using single string input for multiple texts
• Returning single score instead of list
• Using query params for large input lists