Retrieval strategies (similarity, MMR, hybrid) in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Retrieval strategies (similarity, MMR, hybrid)

Which metric matters for retrieval strategies and WHY

For retrieval strategies like similarity search, Maximal Marginal Relevance (MMR), and hybrid methods, the key metrics are Precision, Recall, and F1-score. These metrics tell us how well the system finds relevant items (Recall) and how accurate those found items are (Precision). Since retrieval aims to balance finding many relevant results without too many irrelevant ones, F1-score helps combine both.

Additionally, metrics like Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) are important because they consider the order of retrieved items, rewarding systems that rank relevant items higher.

Confusion matrix for retrieval

      Retrieved Items
      +-----------------------+
      | Relevant | Not Relevant|
    -----------------------------
    Relevant    |   TP    |    FN    |
    Not Relevant|   FP    |    TN    |
    -----------------------------
    Total samples = TP + FP + TN + FN
    

True Positives (TP): Relevant items correctly retrieved.
False Positives (FP): Irrelevant items retrieved.
False Negatives (FN): Relevant items missed.
True Negatives (TN): Irrelevant items correctly not retrieved.

Precision vs Recall tradeoff with examples

Similarity search often favors high recall to find as many relevant items as possible, even if some irrelevant ones appear (lower precision).

MMR balances relevance and diversity, improving precision by reducing redundant results but may slightly reduce recall.

Hybrid methods combine strategies to optimize both precision and recall, aiming for a better overall F1-score.

Example: In a news search engine, high recall means showing all relevant articles, but too many irrelevant ones annoy users (low precision). MMR helps by showing diverse but relevant articles, improving user satisfaction.

What "good" vs "bad" metric values look like

Good retrieval: Precision and recall both above 0.8, F1-score close to 0.85 or higher, MAP and NDCG near 0.9, meaning most retrieved items are relevant and ranked well.

Bad retrieval: Precision below 0.5 means many irrelevant items retrieved; recall below 0.5 means many relevant items missed. Low F1-score (below 0.6) shows poor balance. MAP and NDCG near 0.5 indicate random or poor ranking.

Common pitfalls in retrieval metrics

• Ignoring diversity: High precision but all results very similar can reduce usefulness.
• Overfitting to training queries: Metrics look great on known queries but fail on new ones.
• Data leakage: Using test data during training inflates metrics falsely.
• Accuracy paradox: Accuracy is not useful here because many items are irrelevant; precision and recall matter more.
• Not considering ranking: Metrics like precision ignore order; use MAP or NDCG to evaluate ranking quality.

Self-check question

Your retrieval model has 98% accuracy but only 12% recall on relevant items. Is it good for production?

Answer: No. The high accuracy is misleading because most items are irrelevant, so the model guesses irrelevant often and looks accurate. But 12% recall means it misses 88% of relevant items, which is very poor. The model fails to find most relevant results, so it is not good for production.

Key Result

Precision, recall, and ranking metrics like MAP and NDCG are key to evaluate retrieval strategies balancing relevance and diversity.

Practice

1. Which retrieval strategy focuses on ranking results purely based on how close they are to the query?

easy

A. Random retrieval

B. Maximal Marginal Relevance (MMR)

C. Similarity-based retrieval

D. Hybrid retrieval

Solution

1. Step 1: Understand similarity-based retrieval

   Similarity-based retrieval ranks results by how close or similar they are to the query, focusing only on relevance.

2. Step 2: Compare with other strategies

   MMR balances relevance and diversity, hybrid combines methods, and random is unrelated.

3. Final Answer:

   Similarity-based retrieval -> Option C

4. Quick Check:

   Similarity = closeness only [OK]

Hint: Similarity means closest match only [OK]

Common Mistakes:

• Confusing MMR with similarity
• Thinking hybrid is only similarity
• Choosing random as a valid strategy

2. Which of the following is the correct way to describe Maximal Marginal Relevance (MMR)?

easy

A. Combines all retrieval methods without weighting

B. Ranks results by random selection

C. Only uses keyword matching

D. Balances relevance and diversity in retrieval

Solution

1. Step 1: Define MMR

   MMR is designed to balance relevance to the query and diversity among the results to avoid redundancy.

2. Step 2: Eliminate incorrect options

   Random selection is unrelated, keyword matching is too narrow, and combining without weighting is not MMR.

3. Final Answer:

   Balances relevance and diversity in retrieval -> Option D

4. Quick Check:

   MMR = relevance + diversity [OK]

Hint: MMR mixes relevance with diversity [OK]

Common Mistakes:

• Thinking MMR is random
• Assuming MMR uses only keywords
• Believing MMR combines methods blindly

3. Given the following pseudo-code for a hybrid retrieval method combining similarity and MMR scores:

results = []
for doc in documents:
    sim_score = similarity(query, doc)
    mmr_score = mmr(query, doc, results)
    combined_score = 0.6 * sim_score + 0.4 * mmr_score
    results.append((doc, combined_score))
results.sort(key=lambda x: x[1], reverse=True)
print([doc for doc, score in results[:3]])

What does this code output?

medium

A. Top 3 documents ranked by combined similarity and MMR scores

B. Top 3 documents ranked by similarity score only

C. Top 3 documents ranked by MMR score only

D. Random 3 documents from the list

Solution

1. Step 1: Analyze score calculation

   The code calculates a combined score using 60% similarity and 40% MMR for each document.

2. Step 2: Understand sorting and output

   Documents are sorted by this combined score in descending order, then top 3 are printed.

3. Final Answer:

   Top 3 documents ranked by combined similarity and MMR scores -> Option A

4. Quick Check:

   Hybrid = combined scores [OK]

Hint: Check weighted sum and sorting for final ranking [OK]

Common Mistakes:

• Ignoring MMR score in combined score
• Assuming sorting by similarity only
• Thinking output is random

4. Consider this buggy code snippet for MMR retrieval:

def mmr(query, docs, selected):
    scores = []
    for doc in docs:
        relevance = similarity(query, doc)
        diversity = min([similarity(doc, s) for s in selected])
        score = relevance - 0.5 * diversity
        scores.append((doc, score))
    return max(scores, key=lambda x: x[1])[0]

What is the main error causing a crash when selected is empty?

medium

A. Using min() on an empty list causes an error

B. Incorrect use of max() function

C. Missing return statement

D. Similarity function is undefined

Solution

1. Step 1: Identify cause of crash

   When selected is empty, the list inside min() is empty, causing a ValueError.

2. Step 2: Understand min() behavior

   min() cannot operate on empty lists, so the code crashes at that line.

3. Final Answer:

   Using min() on an empty list causes an error -> Option A

4. Quick Check:

   min(empty list) = error [OK]

Hint: Check min() on empty lists for errors [OK]

Common Mistakes:

• Blaming max() instead of min()
• Ignoring empty list edge case
• Assuming similarity is undefined

5. You want to improve a search system by combining similarity and MMR retrieval. Which approach best balances relevance and diversity in the final results?

hard

A. Use MMR with a diversity weight of zero

B. Combine similarity and MMR scores with adjustable weights

C. Use only similarity scores to rank results

D. Randomly shuffle results after similarity ranking

Solution

1. Step 1: Understand the goal

   Balancing relevance and diversity requires combining both similarity and MMR scores meaningfully.

2. Step 2: Evaluate options

   Using only similarity or zero diversity weight ignores diversity; random shuffling loses relevance order.

3. Step 3: Best approach

   Combining similarity and MMR with adjustable weights allows tuning the balance effectively.

4. Final Answer:

   Combine similarity and MMR scores with adjustable weights -> Option B

5. Quick Check:

   Hybrid weighted combination = best balance [OK]

Hint: Adjust weights to balance relevance and diversity [OK]

Common Mistakes:

• Ignoring diversity by using similarity only
• Setting diversity weight to zero in MMR
• Randomizing results without scoring