Why similarity measures find related text in NLP - Why Metrics Matter

Metrics & Evaluation - Why similarity measures find related text

Which metric matters and WHY

When we want to find related text, we measure how close or similar two pieces of text are. The key metrics are Cosine Similarity and Jaccard Similarity. Cosine similarity measures the angle between two text vectors, showing how similar their meaning is regardless of length. Jaccard similarity compares shared words or features. These metrics help us find texts that talk about the same ideas or topics.

Confusion matrix or equivalent visualization

Related Text Pairs (Positive) vs Not Related (Negative):

               Predicted Related   Predicted Not Related
Actual Related       TP = 80             FN = 20
Actual Not Related   FP = 15             TN = 85

Total samples = 200

From this:
Precision = TP / (TP + FP) = 80 / (80 + 15) = 0.842
Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.8
F1 Score = 2 * (0.842 * 0.8) / (0.842 + 0.8) ≈ 0.82

This shows how well similarity measures help find truly related text pairs.

Precision vs Recall tradeoff with examples

If we want to find related text, sometimes we want to be very sure the pairs we find are truly related (high precision). For example, in a legal document search, wrong matches waste time.

Other times, we want to find as many related texts as possible (high recall). For example, in research, missing related papers is bad.

Improving precision may lower recall and vice versa. Choosing the right balance depends on the task.

What good vs bad metric values look like

Good: Precision and recall both above 0.8 means most found pairs are truly related and most related pairs are found.

Bad: Precision below 0.5 means many unrelated pairs are marked related. Recall below 0.5 means many related pairs are missed.

For similarity measures, a good threshold to decide relatedness is key to get good precision and recall.

Common pitfalls in metrics

• Accuracy paradox: If most text pairs are unrelated, a model that always says "not related" can have high accuracy but is useless.
• Data leakage: Using the same text in training and testing can inflate similarity scores.
• Overfitting: Tuning similarity thresholds too closely on one dataset may not work on new texts.

Self-check question

Your similarity model finds related text pairs with 98% accuracy but only 12% recall. Is it good for finding related texts? Why or why not?

Answer: No, because it misses most related pairs (low recall). It finds very few related texts even if it is usually correct when it does. For related text search, missing many related pairs is a big problem.

Key Result

Cosine and Jaccard similarity metrics help find related text by measuring closeness; balancing precision and recall is key for good results.