Why Hybrid search (semantic + keyword) in Prompt Engineering / GenAI? - Purpose & Use Cases

The Big Idea

What if your search could understand what you mean, not just what you type?

The Scenario

Imagine you have a huge library of documents and you want to find information about "healthy eating habits." You try to search by typing exact words, but you miss documents that use different phrases or synonyms. Or you try to read through everything manually, which takes forever.

The Problem

Searching only by exact words means you miss relevant info that uses different wording. Reading everything yourself is slow and tiring. You might also get too many unrelated results because keyword search can't understand meaning. This makes finding the right info frustrating and error-prone.

The Solution

Hybrid search combines the best of both worlds: it uses keyword search to catch exact matches and semantic search to understand the meaning behind words. This way, you find documents that are truly relevant, even if they don't use your exact words. It saves time and gives better results.

Before vs After

✗ Before

results = [doc for doc in docs if 'healthy eating' in doc.text]

✓ After

results = hybrid_search(query='healthy eating habits', docs=docs)

What It Enables

Hybrid search lets you quickly find meaningful and precise information from large collections, even when words differ.

Real Life Example

A health app uses hybrid search to help users find recipes and tips that match their goals, even if they type different phrases like "nutritious meals" or "good diet." This makes the app smarter and more helpful.

Key Takeaways

Manual keyword search misses meaning and synonyms.

Reading everything manually is slow and tiring.

Hybrid search finds relevant info by combining meaning and exact words.

Practice

1. What is the main advantage of hybrid search combining semantic and keyword methods?

easy

A. It improves search relevance by using both exact words and meaning.

B. It only uses exact keyword matching for faster results.

C. It ignores word meanings to focus on keyword frequency.

D. It replaces keywords with random words for variety.

Solution

1. Step 1: Understand keyword and semantic search roles

   Keyword search finds exact word matches; semantic search finds meaning matches.

2. Step 2: Combine both for better results

   Hybrid search uses both to improve relevance and user satisfaction.

3. Final Answer:

   It improves search relevance by using both exact words and meaning. -> Option A

4. Quick Check:

   Hybrid search = better relevance [OK]

Hint: Hybrid = exact words + meaning for best results [OK]

Common Mistakes:

• Thinking hybrid search uses only keywords
• Assuming semantic search ignores keywords
• Believing hybrid search slows down search always

2. Which of the following is the correct way to combine semantic and keyword scores in hybrid search?

easy

A. final_score = semantic_score * keyword_score

B. final_score = semantic_score / keyword_score

C. final_score = semantic_score - keyword_score

D. final_score = semantic_score + keyword_score

Solution

1. Step 1: Understand score combination methods

   Adding scores balances contributions from both semantic and keyword parts.

2. Step 2: Choose addition for hybrid scoring

   Adding semantic and keyword scores is common to combine relevance signals.

3. Final Answer:

   final_score = semantic_score + keyword_score -> Option D

4. Quick Check:

   Hybrid score = sum of semantic and keyword [OK]

Hint: Add scores to combine semantic and keyword relevance [OK]

Common Mistakes:

• Multiplying scores causing very small or large values
• Subtracting scores losing positive relevance
• Dividing scores causing errors if denominator is zero

3. Given the code snippet:

semantic_scores = [0.8, 0.5, 0.3]
keyword_scores = [0.6, 0.7, 0.4]
final_scores = [s + k for s, k in zip(semantic_scores, keyword_scores)]
print(final_scores)

What is the output?

medium

A. [1.4, 1.2, 0.7]

B. [0.2, -0.2, -0.1]

C. [0.48, 0.35, 0.12]

D. [1.2, 1.4, 0.7]

Solution

1. Step 1: Add corresponding semantic and keyword scores

   0.8+0.6=1.4, 0.5+0.7=1.2, 0.3+0.4=0.7

2. Step 2: Create list of summed scores

   final_scores = [1.4, 1.2, 0.7]

3. Final Answer:

   [1.4, 1.2, 0.7] -> Option A

4. Quick Check:

   Sum pairs = [1.4, 1.2, 0.7] [OK]

Hint: Add pairs element-wise for final scores [OK]

Common Mistakes:

• Multiplying instead of adding scores
• Mixing order of scores in zip
• Confusing subtraction with addition

4. Identify the error in this hybrid search scoring code:

semantic_scores = [0.9, 0.4, 0.7]
keyword_scores = [0.5, 0.6]
final_scores = [s + k for s, k in zip(semantic_scores, keyword_scores)]
print(final_scores)

medium

A. Adding scores should use multiplication instead.

B. Using zip causes a syntax error here.

C. Lists have different lengths causing missing scores.

D. The print statement is missing parentheses.

Solution

1. Step 1: Check list lengths

   semantic_scores has 3 items; keyword_scores has 2 items.

2. Step 2: Understand zip behavior

   zip stops at shortest list length, so last semantic score is ignored.

3. Final Answer:

   Lists have different lengths causing missing scores. -> Option C

4. Quick Check:

   Unequal list lengths truncate results [OK]

Hint: Ensure lists are same length before zipping [OK]

Common Mistakes:

• Assuming zip pads shorter list automatically
• Thinking zip causes syntax error
• Believing multiplication is required for hybrid scores

5. You want to improve a hybrid search system by weighting semantic similarity twice as much as keyword matching. Which formula correctly applies this?

hard

A. final_score = semantic_score + 2 * keyword_score

B. final_score = 2 * semantic_score + keyword_score

C. final_score = semantic_score * keyword_score * 2

D. final_score = (semantic_score + keyword_score) / 2

Solution

1. Step 1: Identify weighting requirement

   Semantic similarity should count double compared to keyword score.

2. Step 2: Apply weights in formula

   Multiply semantic_score by 2, then add keyword_score.

3. Final Answer:

   final_score = 2 * semantic_score + keyword_score -> Option B

4. Quick Check:

   Semantic weighted double = 2 * semantic + keyword [OK]

Hint: Multiply semantic score by 2 before adding keyword [OK]

Common Mistakes:

• Weighting keyword score instead of semantic
• Multiplying all scores together
• Dividing sum instead of weighting