Sentence-BERT for embeddings in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Sentence-BERT for embeddings

Which metric matters for Sentence-BERT embeddings and WHY

Sentence-BERT creates vector representations (embeddings) of sentences. To check how good these embeddings are, we often use cosine similarity. This measures how close two sentence vectors are, showing if they mean similar things. For tasks like sentence similarity or clustering, cosine similarity helps us see if the model groups related sentences well.

When Sentence-BERT is used for classification or retrieval, metrics like accuracy, precision, and recall become important. These tell us how well the embeddings help the model find or classify the right sentences.

Confusion matrix example for Sentence-BERT in classification

      Actual \ Predicted | Positive | Negative
      ---------------------------------------
      Positive           |   TP=85  |  FN=15
      Negative           |   FP=10  |  TN=90
    

Here, TP means sentences correctly matched as similar, FP means sentences wrongly matched, FN means missed similar sentences, and TN means correctly identified as not similar.

Precision vs Recall tradeoff with Sentence-BERT

Imagine you use Sentence-BERT to find similar customer questions in a help center.

• High precision: Most found questions are truly similar. Good if you want to avoid showing unrelated answers.
• High recall: You find almost all similar questions, even if some are less related. Good if you want to make sure no relevant question is missed.

Choosing between precision and recall depends on your goal. For example, if showing wrong answers is bad, prioritize precision. If missing any related question is bad, prioritize recall.

What good vs bad metric values look like for Sentence-BERT embeddings

• Good: Precision and recall above 0.8, F1 score above 0.8, cosine similarity scores clearly separate similar and dissimilar pairs.
• Bad: Precision or recall below 0.5, F1 score low, cosine similarity scores overlap a lot between similar and dissimilar sentences, making it hard to tell them apart.

Common pitfalls when evaluating Sentence-BERT embeddings

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., many dissimilar pairs).
• Data leakage: Using test sentences seen during training inflates metrics falsely.
• Overfitting: Embeddings work well on training data but poorly on new sentences.
• Ignoring threshold tuning: Cosine similarity needs a good cutoff to decide similarity; wrong thresholds hurt precision and recall.

Self-check question

Your Sentence-BERT model finds similar sentences with 98% accuracy but only 12% recall on similar pairs. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most similar sentences, even if it is usually correct when it does find one. For tasks needing to find all similar sentences, missing many is a big problem.

Key Result

Cosine similarity is key for Sentence-BERT embeddings; precision and recall show how well similar sentences are found.