Retrieval strategies (similarity, MMR, hybrid) in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Retrieval strategies (similarity, MMR, hybrid)

Which metric matters for retrieval strategies and WHY

For retrieval strategies like similarity search, Maximal Marginal Relevance (MMR), and hybrid methods, the key metrics are Precision, Recall, and F1-score. These metrics tell us how well the system finds relevant items (Recall) and how accurate those found items are (Precision). Since retrieval aims to balance finding many relevant results without too many irrelevant ones, F1-score helps combine both.

Additionally, metrics like Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) are important because they consider the order of retrieved items, rewarding systems that rank relevant items higher.

Confusion matrix for retrieval

      Retrieved Items
      +-----------------------+
      | Relevant | Not Relevant|
    -----------------------------
    Relevant    |   TP    |    FN    |
    Not Relevant|   FP    |    TN    |
    -----------------------------
    Total samples = TP + FP + TN + FN
    

True Positives (TP): Relevant items correctly retrieved.
False Positives (FP): Irrelevant items retrieved.
False Negatives (FN): Relevant items missed.
True Negatives (TN): Irrelevant items correctly not retrieved.

Precision vs Recall tradeoff with examples

Similarity search often favors high recall to find as many relevant items as possible, even if some irrelevant ones appear (lower precision).

MMR balances relevance and diversity, improving precision by reducing redundant results but may slightly reduce recall.

Hybrid methods combine strategies to optimize both precision and recall, aiming for a better overall F1-score.

Example: In a news search engine, high recall means showing all relevant articles, but too many irrelevant ones annoy users (low precision). MMR helps by showing diverse but relevant articles, improving user satisfaction.

What "good" vs "bad" metric values look like

Good retrieval: Precision and recall both above 0.8, F1-score close to 0.85 or higher, MAP and NDCG near 0.9, meaning most retrieved items are relevant and ranked well.

Bad retrieval: Precision below 0.5 means many irrelevant items retrieved; recall below 0.5 means many relevant items missed. Low F1-score (below 0.6) shows poor balance. MAP and NDCG near 0.5 indicate random or poor ranking.

Common pitfalls in retrieval metrics

• Ignoring diversity: High precision but all results very similar can reduce usefulness.
• Overfitting to training queries: Metrics look great on known queries but fail on new ones.
• Data leakage: Using test data during training inflates metrics falsely.
• Accuracy paradox: Accuracy is not useful here because many items are irrelevant; precision and recall matter more.
• Not considering ranking: Metrics like precision ignore order; use MAP or NDCG to evaluate ranking quality.

Self-check question

Your retrieval model has 98% accuracy but only 12% recall on relevant items. Is it good for production?

Answer: No. The high accuracy is misleading because most items are irrelevant, so the model guesses irrelevant often and looks accurate. But 12% recall means it misses 88% of relevant items, which is very poor. The model fails to find most relevant results, so it is not good for production.

Key Result

Precision, recall, and ranking metrics like MAP and NDCG are key to evaluate retrieval strategies balancing relevance and diversity.