Hybrid search strategies in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Hybrid search strategies

Which metric matters for Hybrid Search Strategies and WHY

Hybrid search combines two ways to find answers: exact matching and smart guessing. The key metrics are Recall and Precision. Recall shows how many good answers the search finds out of all possible good answers. Precision shows how many found answers are actually good. We want high recall so we don't miss useful results, and high precision so results are relevant and not noisy.

Confusion Matrix for Hybrid Search Results

      | Predicted Relevant | Predicted Irrelevant |
      |--------------------|---------------------|
      | True Positive (TP)  | False Positive (FP)  |
      | False Negative (FN) | True Negative (TN)   |

      TP: Good results found
      FP: Wrong results shown
      FN: Good results missed
      TN: Correctly ignored bad results

      Total samples = TP + FP + FN + TN
    

Precision vs Recall Tradeoff with Examples

Imagine searching for a recipe. If you want to see every possible recipe (high recall), you might get many unrelated ones (lower precision). If you want only the best matches (high precision), you might miss some good recipes (lower recall). Hybrid search tries to balance this by combining exact matches (high precision) and semantic matches (high recall).

For example, in a legal document search, missing a relevant case (low recall) can be costly, so recall is more important. In a product search, showing too many unrelated items (low precision) frustrates users, so precision is key.

What Good vs Bad Metric Values Look Like

Good: Precision and recall both above 0.8 means the search finds most relevant results and keeps irrelevant ones low.

Bad: Precision below 0.5 means many wrong results show up. Recall below 0.5 means many good results are missed.

For hybrid search, a good balance is key. For example, precision = 0.85 and recall = 0.75 is usually better than precision = 0.95 but recall = 0.3.

Common Pitfalls in Hybrid Search Metrics

• Accuracy paradox: High accuracy can be misleading if most data is irrelevant. For example, if 95% of documents are irrelevant, a model that always says "irrelevant" has 95% accuracy but is useless.
• Data leakage: If test data leaks into training, metrics look better but don't reflect real performance.
• Overfitting: The search may work well on known queries but fail on new ones, showing high precision and recall only on training data.
• Ignoring user intent: Metrics don't capture if results satisfy the user's real need, so qualitative feedback is also important.

Self Check: Your model has 98% accuracy but 12% recall on relevant results. Is it good?

No, it is not good. The model finds very few relevant results (low recall), even if overall accuracy looks high because most data is irrelevant. This means many useful answers are missed, which defeats the purpose of search. Improving recall is critical.

Key Result

Hybrid search needs a good balance of high recall and precision to find relevant results without too much noise.

Practice

1.

What is the main benefit of using a hybrid search strategy in AI?

easy

A. It relies solely on embedding similarity for accuracy.

B. It uses only keyword matching for faster results.

C. It combines different search methods to improve results.

D. It avoids using any search algorithms.

Solution

1. Step 1: Understand hybrid search purpose

   Hybrid search mixes different search methods to get better results than using one method alone.

2. Step 2: Compare options

   It combines different search methods to improve results. correctly states the benefit. The other options either describe single-method approaches or are incorrect.

3. Final Answer:

   It combines different search methods to improve results. -> Option C

4. Quick Check:

   Hybrid search = mix methods [OK]

Hint: Hybrid means mixing methods for better results [OK]

Common Mistakes:

• Thinking hybrid means using only one search method
• Confusing hybrid search with keyword-only search
• Ignoring the benefit of combining methods

2.

Which of the following is the correct way to combine keyword and embedding search scores in a hybrid search?

final_score = ?

easy

A. final_score = 0.5 * keyword_score + 0.5 * embedding_score

B. final_score = keyword_score * embedding_score

C. final_score = max(keyword_score, embedding_score)

D. final_score = keyword_score - embedding_score

Solution

1. Step 1: Understand score combination

   Hybrid search often combines scores by weighted sum to balance keyword and embedding contributions.

2. Step 2: Evaluate options

   final_score = 0.5 * keyword_score + 0.5 * embedding_score uses weighted sum, which is common. Multiplying scores can distort results. Taking the max ignores combined info. Subtracting can give negative scores.

3. Final Answer:

   final_score = 0.5 * keyword_score + 0.5 * embedding_score -> Option A

4. Quick Check:

   Weighted sum combines scores [OK]

Hint: Use weighted sum to combine scores in hybrid search [OK]

Common Mistakes:

• Multiplying scores causing skewed results
• Using max ignores combined info
• Subtracting scores can produce negatives

3.

Given the following Python code snippet for hybrid search scoring, what is the output?

keyword_scores = [0.8, 0.6, 0.9]
embedding_scores = [0.7, 0.9, 0.5]
final_scores = [0.5 * k + 0.5 * e for k, e in zip(keyword_scores, embedding_scores)]
print(final_scores)

medium

A. [0.8, 0.9, 0.5]

B. [0.75, 0.75, 0.7]

C. [0.56, 0.54, 0.7]

D. [1.5, 1.5, 1.4]

Solution

1. Step 1: Calculate each final score

   For each pair: (0.8+0.7)/2=0.75, (0.6+0.9)/2=0.75, (0.9+0.5)/2=0.7

2. Step 2: Verify output list

   The list is [0.75, 0.75, 0.7], matching [0.75, 0.75, 0.7].

3. Final Answer:

   [0.75, 0.75, 0.7] -> Option B

4. Quick Check:

   Average scores = [0.75, 0.75, 0.7] [OK]

Hint: Average keyword and embedding scores for final score [OK]

Common Mistakes:

• Adding scores without dividing by 2
• Mixing order of scores
• Printing original scores instead of combined

4.

Identify the error in this hybrid search score calculation code and select the fix:

keyword_scores = [0.9, 0.7]
embedding_scores = [0.6]
final_scores = [0.5 * k + 0.5 * e for k, e in zip(keyword_scores, embedding_scores)]
print(final_scores)

medium

A. No error; code runs fine.

B. Use '+' instead of '*' in score calculation.

C. Replace zip with map to fix length mismatch.

D. Lists have different lengths; use min length or pad shorter list.

Solution

1. Step 1: Check list lengths

   keyword_scores has 2 elements, embedding_scores has 1 element, causing zip to truncate to 1 element.

2. Step 2: Fix length mismatch

   Lists have different lengths; use min length or pad shorter list. suggests using min length or padding shorter list to avoid losing data.

3. Final Answer:

   Lists have different lengths; use min length or pad shorter list. -> Option D

4. Quick Check:

   Length mismatch needs handling [OK]

Hint: Check list lengths before zipping in hybrid search [OK]

Common Mistakes:

• Ignoring length mismatch causing data loss
• Changing operators incorrectly
• Assuming zip auto-fills missing values

5.

You want to build a hybrid search system that first filters documents by keywords, then reranks them by embedding similarity. Which approach best fits this goal?

hard

A. Filter documents by keywords, then rerank filtered set by embedding similarity.

B. Run embedding search first, then filter results by keywords.

C. Combine keyword and embedding scores equally on all documents without filtering.

D. Use only keyword search for filtering and ignore embeddings.

Solution

1. Step 1: Understand filtering and reranking

   Filtering by keywords narrows down documents quickly; reranking by embeddings improves relevance.

2. Step 2: Match approach to goal

   Filter documents by keywords, then rerank filtered set by embedding similarity. matches the goal: filter first, then rerank. Run embedding search first, then filter results by keywords. reverses order, less efficient. Combine keyword and embedding scores equally on all documents without filtering. skips filtering, less efficient. Use only keyword search for filtering and ignore embeddings. ignores embeddings, losing semantic power.

3. Final Answer:

   Filter documents by keywords, then rerank filtered set by embedding similarity. -> Option A

4. Quick Check:

   Filter then rerank = best hybrid approach [OK]

Hint: Filter first, rerank second for efficient hybrid search [OK]

Common Mistakes:

• Reranking before filtering wastes resources
• Ignoring filtering step reduces speed
• Using only one method loses hybrid benefits