LSTM for text in NLP - Model Metrics & Evaluation

Metrics & Evaluation - LSTM for text

Which metric matters for LSTM text models and WHY

For LSTM models working with text, the main goal is often to correctly predict sequences or classify text. Common metrics include accuracy for classification tasks, and perplexity or cross-entropy loss for language modeling. Accuracy tells us how many text samples were correctly labeled. Perplexity measures how well the model predicts the next word, with lower values meaning better predictions. These metrics help us understand if the model is learning meaningful patterns in text.

Confusion matrix example for text classification

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 80 | False Negative (FN): 20 |
      | False Positive (FP): 10 | True Negative (TN): 90 |

      Total samples = TP + FP + TN + FN = 80 + 10 + 90 + 20 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
      Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
    

Precision vs Recall tradeoff with examples

In text tasks, the balance between precision and recall depends on the goal:

• Spam detection: High precision is important. We want to avoid marking good emails as spam (false positives).
• Sentiment analysis for customer feedback: High recall is important. We want to catch as many negative comments as possible, even if some are missed.

LSTM models can be tuned to favor precision or recall by adjusting thresholds or loss functions.

What good vs bad metric values look like for LSTM text models

Good metrics for text classification with LSTM:

• Accuracy above 85% on balanced data
• Precision and recall both above 80%
• F1 score close to precision and recall

Bad metrics might be:

• Accuracy near random chance (e.g., 50% for binary)
• Very high precision but very low recall (or vice versa), showing imbalance
• High loss or perplexity in language modeling, indicating poor prediction

Common pitfalls in evaluating LSTM text models

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., 90% accuracy by always predicting the majority class).
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting: Very low training loss but high test loss means the model memorizes training text but fails on new text.
• Ignoring class imbalance: Not using metrics like F1 or balanced accuracy can hide poor performance on minority classes.

Self-check question

Your LSTM text classification model has 98% accuracy but only 12% recall on the positive class (e.g., spam). Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is likely due to many negative samples dominating the data. The very low recall means the model misses most positive cases (spam), which is critical to catch. This model would fail to identify most spam emails, making it unreliable in practice.

Key Result

For LSTM text models, balanced precision and recall with high accuracy and low loss indicate good performance.