Document similarity ranking in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Document similarity ranking

Which metric matters for Document similarity ranking and WHY

For document similarity ranking, the key metrics are Mean Reciprocal Rank (MRR), Precision@k, and Recall@k. These metrics measure how well the model ranks relevant documents near the top of the list.

MRR tells us how quickly the first relevant document appears in the ranked list. A higher MRR means users find what they want faster.

Precision@k measures the fraction of relevant documents in the top k results. It shows how accurate the top results are.

Recall@k measures how many of all relevant documents appear in the top k. It shows how complete the top results are.

These metrics matter because users usually look at only the first few results. Good ranking means relevant documents appear early.

Confusion matrix or equivalent visualization

Document similarity ranking does not use a traditional confusion matrix because it is a ranking task, not a simple yes/no classification.

Instead, we look at ranked lists and check positions of relevant documents.

Query: "Climate change impact"
Ranked Documents:
1. "Climate change effects on oceans" (Relevant)
2. "Sports news today" (Not relevant)
3. "Global warming and weather" (Relevant)
4. "Cooking recipes" (Not relevant)

Metrics:
- MRR = 1 / 1 = 1.0 (first relevant at rank 1)
- Precision@3 = 2 relevant / 3 total = 0.67
- Recall@3 = 2 relevant found / 3 total relevant = 0.67
    

Precision vs Recall tradeoff with concrete examples

In document similarity ranking, precision means how many of the top results are actually relevant. Recall means how many relevant documents are found in the top results.

Example 1: High precision, low recall
The top 3 results are all relevant, but there are 10 relevant documents total. The user sees only a few relevant documents but they are all correct.

Example 2: High recall, low precision
The top 13 results include 8 relevant documents but also 5 irrelevant ones. The user sees most relevant documents but mixed with noise.

Depending on the use case, you might want to prioritize precision (show only very relevant docs) or recall (show as many relevant docs as possible).

What "good" vs "bad" metric values look like for this use case

Good values:

• MRR close to 1.0 (first relevant document appears at top rank)
• Precision@5 above 0.8 (at least 4 out of 5 top documents are relevant)
• Recall@10 above 0.7 (most relevant documents appear in top 10)

Bad values:

• MRR below 0.3 (relevant documents appear very late)
• Precision@5 below 0.4 (more than half of top results are irrelevant)
• Recall@10 below 0.3 (most relevant documents are missing from top results)

Metrics pitfalls

• Ignoring ranking order: Treating document similarity as binary classification loses ranking info.
• Data leakage: Using test documents in training can inflate metrics falsely.
• Overfitting: Model memorizes training documents, performs poorly on new queries.
• Unbalanced relevance: Few relevant documents per query can make metrics unstable.
• Using accuracy: Accuracy is not meaningful for ranking tasks.

Self-check question

Your document similarity model has an MRR of 0.95 but Precision@5 of 0.3. Is it good?

Answer: The model finds a relevant document very quickly (high MRR), but many of the top 5 results are irrelevant (low precision). This means users find one relevant document fast but see many irrelevant ones too. Depending on the use case, this might be okay or need improvement to increase precision.

Key Result

Mean Reciprocal Rank (MRR), Precision@k, and Recall@k are key metrics to evaluate how well relevant documents appear early in the ranked list.

Practice

1. What does document similarity ranking help us do in natural language processing?

easy

A. Find how related two texts are based on their content

B. Translate documents into different languages

C. Summarize long documents into short ones

D. Detect spelling errors in documents

Solution

1. Step 1: Understand the purpose of document similarity ranking

   Document similarity ranking is used to compare texts and find how closely related they are based on their content.

2. Step 2: Identify the correct description

   Among the options, only finding relatedness of texts matches the purpose of document similarity ranking.

3. Final Answer:

   Find how related two texts are based on their content -> Option A

4. Quick Check:

   Document similarity ranking = Find related texts [OK]

Hint: Think: similarity means how close or related texts are [OK]

Common Mistakes:

• Confusing similarity ranking with translation
• Thinking it summarizes documents
• Mixing it up with spell checking

2. Which of the following is the correct way to compute cosine similarity between two vectors A and B in Python using NumPy?

easy

A. np.dot(A, B) * (np.linalg.norm(A) + np.linalg.norm(B))

B. np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B))

C. np.dot(A, B) - (np.linalg.norm(A) * np.linalg.norm(B))

D. np.dot(A, B) / (np.linalg.norm(A) + np.linalg.norm(B))

Solution

1. Step 1: Recall cosine similarity formula

   Cosine similarity = dot product of vectors divided by product of their magnitudes (norms).

2. Step 2: Match formula to code

   np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B)) correctly implements this formula using np.dot and np.linalg.norm.

3. Final Answer:

   np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B)) -> Option B

4. Quick Check:

   Cosine similarity formula = np.dot(A, B) / (np.linalg.norm(A) * np.linalg.norm(B)) [OK]

Hint: Cosine similarity = dot product ÷ (norm A x norm B) [OK]

Common Mistakes:

• Adding norms instead of multiplying
• Subtracting norms instead of dividing
• Multiplying dot product by sum of norms

3. Given the following Python code using TF-IDF and cosine similarity, what will be the printed similarity score between the two documents?

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ['apple orange banana', 'banana fruit apple']
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(docs)
sim_score = cosine_similarity(X[0], X[1])[0][0]
print(round(sim_score, 2))

medium

A. 0.50

B. 1.00

C. 0.58

D. 0.00

Solution

1. Step 1: Understand TF-IDF vectorization of similar documents

   Both documents share words 'apple' and 'banana' and have similar content, so their TF-IDF vectors will be close.

2. Step 2: Calculate cosine similarity between vectors

   Cosine similarity between these vectors will be high but less than 1, approximately 0.58 after rounding.

3. Final Answer:

   0.58 -> Option C

4. Quick Check:

   Similarity of similar docs ≈ 0.58 [OK]

Hint: Similar docs have cosine similarity close to 1 but not exactly 1 [OK]

Common Mistakes:

• Assuming similarity is exactly 1 for similar texts
• Confusing cosine similarity with Euclidean distance
• Ignoring TF-IDF weighting effects

4. The following code attempts to compute cosine similarity between two documents but raises an error. What is the main issue?

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ['cat dog', 'dog mouse']
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(docs).toarray()
sim_score = cosine_similarity(X[0], X[1])
print(sim_score)

medium

A. cosine_similarity expects 2D arrays, but X[0] and X[1] are 1D arrays

B. TfidfVectorizer cannot process documents with different words

C. cosine_similarity requires dense arrays, not sparse matrices

D. The print statement has a typo in variable name

Solution

1. Step 1: Check input types for cosine_similarity

   cosine_similarity expects 2D arrays, but X[0] and X[1] are 1D arrays (shape (n_features,)).

2. Step 2: Understand how to fix the error

   Use X[0:1] and X[1:2] or reshape them properly to avoid the error.

3. Final Answer:

   cosine_similarity expects 2D arrays, but X[0] and X[1] are 1D arrays -> Option A

4. Quick Check:

   cosine_similarity input shape = 2D arrays [OK]

Hint: cosine_similarity needs 2D arrays, not single vectors [OK]

Common Mistakes:

• Thinking TfidfVectorizer fails on different words
• Thinking cosine_similarity accepts 1D arrays
• Overlooking variable name typos

5. You have a collection of 3 documents: ['apple banana', 'banana orange', 'apple orange banana']. You want to rank these documents by similarity to the query 'banana apple'. Which approach correctly ranks them from most to least similar using TF-IDF and cosine similarity?

hard

A. Use raw word counts without TF-IDF, rank by Euclidean distance ascending

B. Count word overlaps between query and documents, rank by overlap count ascending

C. Compute TF-IDF vectors but rank by cosine similarity scores ascending

D. Compute TF-IDF vectors for all documents and query, then rank by cosine similarity scores descending

Solution

1. Step 1: Understand ranking by similarity

   To rank documents by similarity to a query, compute vector representations and measure similarity scores, then sort descending (highest similarity first).

2. Step 2: Identify correct method

   TF-IDF vectors and cosine similarity are standard; ranking by descending cosine similarity scores is correct.

3. Final Answer:

   Compute TF-IDF vectors for all documents and query, then rank by cosine similarity scores descending -> Option D

4. Quick Check:

   Similarity ranking = cosine similarity descending [OK]

Hint: Rank documents by highest cosine similarity to query [OK]

Common Mistakes:

• Ranking by ascending similarity (lowest first)
• Using raw counts without weighting
• Ranking by overlap count ascending