Semantic similarity with embeddings in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Semantic similarity with embeddings

Which metric matters for semantic similarity and WHY

For semantic similarity using embeddings, the key metric is cosine similarity. This measures how close two vectors point in the same direction, regardless of their length. It tells us how similar two pieces of text are in meaning.

Why cosine similarity? Because embeddings are numeric vectors representing meaning, and cosine similarity captures the angle between them, which reflects semantic closeness well.

Sometimes, we also use Euclidean distance or Manhattan distance, but cosine similarity is most common and intuitive for meaning comparison.

Confusion matrix or equivalent visualization

Semantic similarity is often a continuous score, not a classification, so confusion matrices don't directly apply. But if we set a threshold to decide if two texts are "similar" or "not similar," we can create a confusion matrix:

      | Predicted Similar | Predicted Not Similar |
      |-------------------|-----------------------|
      | True Positive (TP) | False Positive (FP)    |
      | False Negative (FN)| True Negative (TN)     |
    

For example, if cosine similarity > 0.8 means "similar," then:

• TP: Pairs correctly identified as similar
• FP: Pairs incorrectly identified as similar
• FN: Pairs incorrectly identified as not similar
• TN: Pairs correctly identified as not similar

Precision vs Recall tradeoff with examples

When deciding if two texts are similar, precision and recall matter:

• Precision: Of all pairs predicted similar, how many truly are? High precision means few false alarms.
• Recall: Of all truly similar pairs, how many did we find? High recall means we miss few true matches.

Example: In a plagiarism detector, high recall is important to catch all copied texts, even if some false alarms happen (lower precision).

In a recommendation system, high precision is important to avoid suggesting irrelevant items, even if some good matches are missed (lower recall).

What "good" vs "bad" metric values look like

Good semantic similarity results have:

• Cosine similarity close to 1 for truly similar pairs (e.g., > 0.8)
• Cosine similarity close to 0 or negative for unrelated pairs
• High precision and recall if thresholding is used (e.g., both > 0.8)

Bad results show:

• High similarity scores for unrelated pairs (false positives)
• Low similarity scores for truly similar pairs (false negatives)
• Precision or recall very low (e.g., < 0.5), meaning many mistakes

Common pitfalls in semantic similarity metrics

• Ignoring context: Embeddings may not capture subtle meaning differences if context is missing.
• Threshold choice: Picking a bad similarity threshold can cause many false positives or negatives.
• Data leakage: Using test data in training embeddings inflates similarity scores unfairly.
• Overfitting: Embeddings tuned too closely to training data may not generalize well.
• Using only accuracy: Accuracy is less meaningful for similarity tasks without clear classes.

Self-check question

Your semantic similarity model has an average cosine similarity of 0.95 on similar pairs but 0.6 on unrelated pairs. Is this good?

Answer: Not really. While 0.95 on similar pairs is excellent, 0.6 on unrelated pairs is quite high, meaning many unrelated pairs appear similar. This can cause many false positives if thresholding is used. You should improve the model to lower similarity scores for unrelated pairs.

Key Result

Cosine similarity is key for semantic similarity; good models show high similarity for related pairs and low for unrelated pairs.