Document similarity ranking in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Document similarity ranking

Which metric matters for Document similarity ranking and WHY

For document similarity ranking, the key metrics are Mean Reciprocal Rank (MRR), Precision@k, and Recall@k. These metrics measure how well the model ranks relevant documents near the top of the list.

MRR tells us how quickly the first relevant document appears in the ranked list. A higher MRR means users find what they want faster.

Precision@k measures the fraction of relevant documents in the top k results. It shows how accurate the top results are.

Recall@k measures how many of all relevant documents appear in the top k. It shows how complete the top results are.

These metrics matter because users usually look at only the first few results. Good ranking means relevant documents appear early.

Confusion matrix or equivalent visualization

Document similarity ranking does not use a traditional confusion matrix because it is a ranking task, not a simple yes/no classification.

Instead, we look at ranked lists and check positions of relevant documents.

Query: "Climate change impact"
Ranked Documents:
1. "Climate change effects on oceans" (Relevant)
2. "Sports news today" (Not relevant)
3. "Global warming and weather" (Relevant)
4. "Cooking recipes" (Not relevant)

Metrics:
- MRR = 1 / 1 = 1.0 (first relevant at rank 1)
- Precision@3 = 2 relevant / 3 total = 0.67
- Recall@3 = 2 relevant found / 3 total relevant = 0.67
    

Precision vs Recall tradeoff with concrete examples

In document similarity ranking, precision means how many of the top results are actually relevant. Recall means how many relevant documents are found in the top results.

Example 1: High precision, low recall
The top 3 results are all relevant, but there are 10 relevant documents total. The user sees only a few relevant documents but they are all correct.

Example 2: High recall, low precision
The top 13 results include 8 relevant documents but also 5 irrelevant ones. The user sees most relevant documents but mixed with noise.

Depending on the use case, you might want to prioritize precision (show only very relevant docs) or recall (show as many relevant docs as possible).

What "good" vs "bad" metric values look like for this use case

Good values:

• MRR close to 1.0 (first relevant document appears at top rank)
• Precision@5 above 0.8 (at least 4 out of 5 top documents are relevant)
• Recall@10 above 0.7 (most relevant documents appear in top 10)

Bad values:

• MRR below 0.3 (relevant documents appear very late)
• Precision@5 below 0.4 (more than half of top results are irrelevant)
• Recall@10 below 0.3 (most relevant documents are missing from top results)

Metrics pitfalls

• Ignoring ranking order: Treating document similarity as binary classification loses ranking info.
• Data leakage: Using test documents in training can inflate metrics falsely.
• Overfitting: Model memorizes training documents, performs poorly on new queries.
• Unbalanced relevance: Few relevant documents per query can make metrics unstable.
• Using accuracy: Accuracy is not meaningful for ranking tasks.

Self-check question

Your document similarity model has an MRR of 0.95 but Precision@5 of 0.3. Is it good?

Answer: The model finds a relevant document very quickly (high MRR), but many of the top 5 results are irrelevant (low precision). This means users find one relevant document fast but see many irrelevant ones too. Depending on the use case, this might be okay or need improvement to increase precision.

Key Result

Mean Reciprocal Rank (MRR), Precision@k, and Recall@k are key metrics to evaluate how well relevant documents appear early in the ranked list.