Encoder-decoder with attention in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Encoder-decoder with attention

Which metric matters for Encoder-decoder with attention and WHY

For encoder-decoder models with attention, especially in tasks like translation or summarization, BLEU score and ROUGE score are key. They measure how close the model's output is to human references. However, these are not perfect, so perplexity is also used during training to see how well the model predicts the next word. Lower perplexity means the model is more confident and accurate in its predictions.

In classification tasks using encoder-decoder, accuracy, precision, and recall matter to understand how well the model identifies correct outputs.

Confusion matrix example for classification with encoder-decoder

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 50  | False Negative (FN) = 10 |
      | False Positive (FP) = 5  | True Negative (TN) = 35  |

      Total samples = 50 + 10 + 5 + 35 = 100

      Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91
      Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87
    

This matrix helps us see where the model makes mistakes and how precise and complete its predictions are.

Precision vs Recall tradeoff with Encoder-decoder attention

Imagine a translation app using encoder-decoder with attention. If it focuses on precision, it avoids wrong translations but might miss some correct phrases (low recall). If it focuses on recall, it tries to translate everything but may include errors (low precision).

For example, in medical report summarization, high recall is important to not miss critical info, even if some details are less precise. In chatbots, high precision is better to avoid confusing answers.

What good vs bad metric values look like for Encoder-decoder with attention

• Good BLEU/ROUGE: Scores above 0.6 show the model's output closely matches human text.
• Good perplexity: Lower values (e.g., below 20) mean the model predicts words well.
• Good precision and recall: Above 0.8 means the model is accurate and complete in predictions.
• Bad values: BLEU/ROUGE below 0.3, high perplexity (above 50), or precision/recall below 0.5 indicate poor model performance.

Common pitfalls in metrics for Encoder-decoder with attention

• Overfitting: Very low training loss but poor BLEU on test means model memorizes training data, not generalizing.
• Ignoring context: BLEU and ROUGE don't capture meaning well; a sentence can score low but still be good.
• Data leakage: If test data is similar to training, metrics look better but model fails in real use.
• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.

Self-check question

Your encoder-decoder model with attention has 98% accuracy but only 12% recall on the important class (e.g., fraud detection). Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most of the important cases, which is dangerous. High accuracy is misleading because the important class is rare. You need to improve recall to catch more true positives.

Key Result

For encoder-decoder with attention, BLEU/ROUGE and perplexity measure output quality, while precision and recall reveal prediction balance; high recall is crucial in critical tasks.