Information retrieval basics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Information retrieval basics

Which metric matters for Information Retrieval and WHY

In information retrieval, the main goal is to find relevant documents from a large collection based on a user's query. The key metrics are Precision and Recall. Precision tells us how many of the retrieved documents are actually relevant. Recall tells us how many of the relevant documents we managed to find. Both matter because we want to find as many relevant documents as possible (high recall) but also avoid showing irrelevant ones (high precision). The F1 score balances precision and recall into one number. Sometimes, Mean Average Precision (MAP) or Normalized Discounted Cumulative Gain (NDCG) are used to measure ranking quality, but precision and recall are the basics.

Confusion Matrix for Information Retrieval

                | Retrieved Relevant | Retrieved Irrelevant |
----------------|--------------------|----------------------|
Relevant Docs   | True Positives (TP) | False Negatives (FN) |
Irrelevant Docs | False Positives (FP)| True Negatives (TN)  |

Example: Suppose we have 100 documents. 30 are relevant to the query. The system retrieves 40 documents, 25 of which are relevant (TP=25), and 15 are irrelevant (FP=15). The system misses 5 relevant documents (FN=5). The rest 55 documents are irrelevant and not retrieved (TN=55).

Precision vs Recall Tradeoff with Examples

Imagine a search engine. If it shows only a few documents that it is very sure about, precision is high but recall is low because many relevant documents are missed. If it shows many documents including less certain ones, recall is high but precision drops because more irrelevant documents appear.

Example 1: Medical research paper search
Recall is more important. Missing a relevant paper could mean missing critical information.

Example 2: Shopping site search
Precision is more important. Showing irrelevant products annoys users.

What Good vs Bad Metric Values Look Like

Good: Precision and recall both above 0.8 means the system finds most relevant documents and keeps irrelevant ones low.

Bad: Precision below 0.5 means many irrelevant documents are shown. Recall below 0.5 means many relevant documents are missed.

F1 score below 0.6 usually indicates poor balance.

Common Pitfalls in Information Retrieval Metrics

• Accuracy paradox: Accuracy is not useful because most documents are irrelevant, so a system that retrieves nothing can have high accuracy but zero recall.
• Ignoring ranking: Metrics like precision ignore the order of documents, but users care about top results.
• Data leakage: Using test queries or documents in training can inflate metrics falsely.
• Overfitting: Optimizing too much for training queries can reduce generalization to new queries.

Self Check

Your information retrieval system has 98% accuracy but only 12% recall on relevant documents. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most documents are irrelevant. The very low recall means the system misses most relevant documents, which defeats the purpose of retrieval.

Key Result

Precision and recall are key metrics in information retrieval to balance finding relevant documents and avoiding irrelevant ones.