Model serving for NLP - Model Metrics & Evaluation

Metrics & Evaluation - Model serving for NLP

Which metric matters for Model Serving in NLP and WHY

When serving NLP models, the key metrics depend on the task. For classification tasks like sentiment analysis, accuracy, precision, and recall matter because they show how well the model predicts correct labels.

For generation tasks like chatbots, metrics like response time and latency are critical to ensure users get quick answers.

Overall, latency and throughput measure how fast and how many requests the model can handle, which is vital for a smooth user experience.

Confusion Matrix Example for NLP Classification

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 80 | False Positive (FP): 10 |
      | False Negative (FN): 20 | True Negative (TN): 90 |

      Total samples = 80 + 20 + 10 + 90 = 200
    

From this matrix:

• Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
• Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
• Accuracy = (TP + TN) / Total = (80 + 90) / 200 = 0.85
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84

Precision vs Recall Tradeoff in NLP Model Serving

Imagine a spam detection NLP model served to filter emails:

• High Precision: Few good emails are wrongly marked as spam. Users don't miss important emails.
• High Recall: Most spam emails are caught, but some good emails might be wrongly flagged.

For serving, if users complain about missing emails, prioritize precision. If spam floods inboxes, prioritize recall.

Balancing these depends on user needs and model tuning during serving.

Good vs Bad Metric Values for NLP Model Serving

Good metrics:

• Accuracy above 85% for classification tasks.
• Precision and recall both above 80%, showing balanced performance.
• Latency under 200 milliseconds for real-time NLP services.
• Throughput high enough to handle expected user requests without delay.

Bad metrics:

• Accuracy below 70%, indicating many wrong predictions.
• Precision very low (e.g., 50%), causing many false alarms.
• Recall very low (e.g., 40%), missing many true cases.
• Latency above 1 second, making the service feel slow.
• Throughput too low, causing request backlogs.

Common Metrics Pitfalls in NLP Model Serving

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., 95% accuracy but model ignores rare classes).
• Data leakage: If test data leaks into training, metrics look better but model fails in real use.
• Overfitting indicators: Very high training accuracy but low serving accuracy means model learned noise, not real patterns.
• Ignoring latency: A model with great accuracy but slow response is bad for serving.
• Not monitoring drift: Model performance can drop over time if input data changes, so metrics must be tracked continuously.

Self-Check Question

Your NLP model serving spam detection has 98% accuracy but only 12% recall on spam emails. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses 88% of spam emails, letting most spam through. High accuracy is misleading because most emails are not spam, so the model just predicts "not spam" often. For spam detection, recall is critical to catch spam, so this model needs improvement before production.

Key Result

In NLP model serving, balancing precision, recall, and latency ensures accurate and fast predictions for a good user experience.

Practice

1. What is the main purpose of model serving in NLP?

easy

A. To visualize model training progress

B. To train NLP models faster

C. To collect more training data

D. To make NLP models available for real-time use

Solution

1. Step 1: Understand model serving concept

   Model serving means making a trained NLP model ready to answer requests instantly.

2. Step 2: Identify the main goal

   The goal is to provide real-time NLP results to apps or users, not training or data collection.

3. Final Answer:

   To make NLP models available for real-time use -> Option D

4. Quick Check:

   Model serving = real-time use [OK]

Hint: Model serving = ready for instant NLP predictions [OK]

Common Mistakes:

• Confusing serving with training
• Thinking serving collects data
• Assuming serving is for visualization

2. Which of the following is the correct way to serve an NLP model using a Python Flask API?

easy

A. import Flask
app = Flask(__name__)
@app.route('/predict')
def predict():
return 'Prediction result'

B. import flask
app = flask()
@app.route('/predict')
def predict():
return 'Prediction result'

C. from flask import Flask
app = Flask(__name__)
@app.route('/predict')
def predict():
return 'Prediction result'

D. from flask import Flask
app = Flask()
@app.route('/predict')
def predict():
return 'Prediction result'

Solution

1. Step 1: Check Flask import and app creation

   Correct import is from flask import Flask and app created by Flask(__name__).

2. Step 2: Verify route decorator and function

   Route decorator @app.route('/predict') and function returning string is correct.

3. Final Answer:

   Correct Flask API setup with proper import and app creation -> Option C

4. Quick Check:

   Flask import and app = Flask(__name__) [OK]

Hint: Flask app needs Flask(__name__) and correct import [OK]

Common Mistakes:

• Using wrong Flask import syntax
• Missing __name__ in Flask()
• Incorrect app creation call

3. Given this Flask code snippet serving an NLP sentiment model, what will be the output when accessing /predict?text=happy?

from flask import Flask, request, jsonify
app = Flask(__name__)

@app.route('/predict')
def predict():
    text = request.args.get('text')
    if 'happy' in text:
        sentiment = 'positive'
    else:
        sentiment = 'neutral'
    return jsonify({'sentiment': sentiment})

medium

A. {"sentiment": "positive"}

B. {"sentiment": "neutral"}

C. Error: Missing text parameter

D. 404 Not Found

Solution

1. Step 1: Extract query parameter 'text'

   The URL provides text='happy', so text variable is 'happy'.

2. Step 2: Check condition for sentiment

   Since 'happy' is in text, sentiment is set to 'positive'.

3. Final Answer:

   {"sentiment": "positive"} -> Option A

4. Quick Check:

   Text contains 'happy' -> positive sentiment [OK]

Hint: Check if 'happy' in text to decide sentiment [OK]

Common Mistakes:

• Assuming neutral sentiment for 'happy'
• Forgetting to pass text parameter
• Confusing JSON string with Python dict

4. This Flask code for serving an NLP model throws an error. What is the bug?

from flask import Flask, request, jsonify
app = Flask(__name__)

@app.route('/predict')
def predict():
    text = request.args['text']
    sentiment = 'positive' if 'good' in text else 'negative'
    return jsonify(sentiment=sentiment)

if __name__ == '__main__':
    app.run()

medium

A. Missing return statement in predict function

B. Using request.args['text'] causes KeyError if 'text' missing

C. Flask app is not created properly

D. jsonify() cannot accept keyword arguments

Solution

1. Step 1: Analyze request.args usage

   Using request.args['text'] raises KeyError if 'text' parameter is missing in URL.

2. Step 2: Identify safer alternative

   Using request.args.get('text') avoids error by returning None if missing.

3. Final Answer:

   Using request.args['text'] causes KeyError if 'text' missing -> Option B

4. Quick Check:

   request.args['text'] can cause KeyError [OK]

Hint: Use request.args.get() to avoid KeyError [OK]

Common Mistakes:

• Assuming request.args['text'] always exists
• Thinking jsonify can't take keywords
• Ignoring app creation correctness

5. You want to serve a summarization NLP model that sometimes returns empty summaries for very short texts. How can you improve the serving code to handle this edge case gracefully?

hard

A. Add a check to return the original text if the summary is empty

B. Always return an empty string for short texts

C. Raise an error when summary is empty

D. Ignore short texts and return null

Solution

1. Step 1: Identify the problem with empty summaries

   Empty summaries confuse users and reduce usefulness for short texts.

2. Step 2: Implement fallback logic

   Return the original text if the summary is empty to ensure meaningful output.

3. Final Answer:

   Add a check to return the original text if the summary is empty -> Option A

4. Quick Check:

   Fallback to original text if summary empty [OK]

Hint: Return original text if summary is empty to avoid blanks [OK]

Common Mistakes:

• Returning empty string confuses users
• Raising error breaks serving
• Ignoring short texts causes bad UX