Hybrid search (semantic + keyword) in Prompt Engineering / GenAI - Full Explanation

Introduction

Finding the right information quickly can be tricky when you only rely on exact words or just the meaning behind them. Hybrid search solves this by combining two ways to look for information, making searches smarter and more accurate.

Explanation

Keyword Search

Keyword search looks for exact words or phrases typed by the user. It matches these words directly in documents or data, so it works well when you know the exact terms. However, it can miss relevant results if the words are different but the meaning is similar.

Keyword search finds exact word matches but can miss related ideas with different wording.

Semantic Search

Semantic search understands the meaning behind words, not just the exact terms. It uses techniques like word embeddings to find concepts related to the query, even if the words differ. This helps find relevant information that keyword search might miss.

Semantic search finds results based on meaning, capturing related ideas beyond exact words.

Combining Both Approaches

Hybrid search merges keyword and semantic search to get the best of both worlds. It first looks for exact matches and then expands to related meanings. This combination improves accuracy and relevance, especially in complex or large datasets.

Hybrid search improves results by combining exact word matches with understanding of meaning.

Use Cases and Benefits

Hybrid search is useful in places like online stores, document libraries, and customer support where users may use different words for the same idea. It helps users find what they want faster and with fewer missed results.

Hybrid search helps users find information faster and more accurately across many applications.

Real World Analogy

Imagine looking for a book in a library. Keyword search is like checking the exact title or author name, while semantic search is like asking the librarian for books about a topic, even if you don't know the exact title. Hybrid search uses both methods to find the book quickly.

Keyword Search → Looking up a book by its exact title or author name

Semantic Search → Asking the librarian for books about a topic without knowing exact titles

Combining Both Approaches → Using both the exact title and topic description to find the book faster

Use Cases and Benefits → Finding the right book quickly in a large library with many options

Diagram

Diagram

┌───────────────┐      ┌───────────────┐
│ Keyword Input │─────▶│ Keyword Match │
└───────────────┘      └───────────────┘
         │                      │
         │                      ▼
         │              ┌───────────────┐
         │              │  Semantic     │
         │              │  Understanding│
         │              └───────────────┘
         │                      │
         ▼                      ▼
   ┌───────────────┐      ┌───────────────┐
   │ Exact Matches │◀────▶│ Related Mean- │
   │               │      │ ings Found    │
   └───────────────┘      └───────────────┘
             
             ▼
    ┌─────────────────┐
    │ Combined Results │
    └─────────────────┘

This diagram shows how keyword input leads to exact matches and semantic understanding finds related meanings, which combine to form the final search results.

Key Facts

Keyword Search → Search method matching exact words or phrases typed by the user.

Semantic Search → Search method understanding the meaning behind words to find related concepts.

Hybrid Search → A search approach combining keyword and semantic search for better accuracy.

Word Embeddings → Mathematical representations of words capturing their meanings and relationships.

Search Relevance → How well search results match the user's intent and query.

Common Confusions

Believing keyword search alone is enough for all search needs.

Believing keyword search alone is enough for all search needs. Keyword search misses results when users use different words with the same meaning; semantic search helps fill this gap.

Thinking semantic search ignores exact words.

Thinking semantic search ignores exact words. Semantic search focuses on meaning but does not replace keyword search; hybrid search uses both to improve results.

Summary

Hybrid search combines exact word matching and meaning-based search to improve finding relevant information.

Keyword search looks for exact words, while semantic search understands the meaning behind queries.

Using both methods together helps users find better results faster, especially in complex data.

Practice

1. What is the main advantage of hybrid search combining semantic and keyword methods?

easy

A. It improves search relevance by using both exact words and meaning.

B. It only uses exact keyword matching for faster results.

C. It ignores word meanings to focus on keyword frequency.

D. It replaces keywords with random words for variety.

Solution

1. Step 1: Understand keyword and semantic search roles

   Keyword search finds exact word matches; semantic search finds meaning matches.

2. Step 2: Combine both for better results

   Hybrid search uses both to improve relevance and user satisfaction.

3. Final Answer:

   It improves search relevance by using both exact words and meaning. -> Option A

4. Quick Check:

   Hybrid search = better relevance [OK]

Hint: Hybrid = exact words + meaning for best results [OK]

Common Mistakes:

• Thinking hybrid search uses only keywords
• Assuming semantic search ignores keywords
• Believing hybrid search slows down search always

2. Which of the following is the correct way to combine semantic and keyword scores in hybrid search?

easy

A. final_score = semantic_score * keyword_score

B. final_score = semantic_score / keyword_score

C. final_score = semantic_score - keyword_score

D. final_score = semantic_score + keyword_score

Solution

1. Step 1: Understand score combination methods

   Adding scores balances contributions from both semantic and keyword parts.

2. Step 2: Choose addition for hybrid scoring

   Adding semantic and keyword scores is common to combine relevance signals.

3. Final Answer:

   final_score = semantic_score + keyword_score -> Option D

4. Quick Check:

   Hybrid score = sum of semantic and keyword [OK]

Hint: Add scores to combine semantic and keyword relevance [OK]

Common Mistakes:

• Multiplying scores causing very small or large values
• Subtracting scores losing positive relevance
• Dividing scores causing errors if denominator is zero

3. Given the code snippet:

semantic_scores = [0.8, 0.5, 0.3]
keyword_scores = [0.6, 0.7, 0.4]
final_scores = [s + k for s, k in zip(semantic_scores, keyword_scores)]
print(final_scores)

What is the output?

medium

A. [1.4, 1.2, 0.7]

B. [0.2, -0.2, -0.1]

C. [0.48, 0.35, 0.12]

D. [1.2, 1.4, 0.7]

Solution

1. Step 1: Add corresponding semantic and keyword scores

   0.8+0.6=1.4, 0.5+0.7=1.2, 0.3+0.4=0.7

2. Step 2: Create list of summed scores

   final_scores = [1.4, 1.2, 0.7]

3. Final Answer:

   [1.4, 1.2, 0.7] -> Option A

4. Quick Check:

   Sum pairs = [1.4, 1.2, 0.7] [OK]

Hint: Add pairs element-wise for final scores [OK]

Common Mistakes:

• Multiplying instead of adding scores
• Mixing order of scores in zip
• Confusing subtraction with addition

4. Identify the error in this hybrid search scoring code:

semantic_scores = [0.9, 0.4, 0.7]
keyword_scores = [0.5, 0.6]
final_scores = [s + k for s, k in zip(semantic_scores, keyword_scores)]
print(final_scores)

medium

A. Adding scores should use multiplication instead.

B. Using zip causes a syntax error here.

C. Lists have different lengths causing missing scores.

D. The print statement is missing parentheses.

Solution

1. Step 1: Check list lengths

   semantic_scores has 3 items; keyword_scores has 2 items.

2. Step 2: Understand zip behavior

   zip stops at shortest list length, so last semantic score is ignored.

3. Final Answer:

   Lists have different lengths causing missing scores. -> Option C

4. Quick Check:

   Unequal list lengths truncate results [OK]

Hint: Ensure lists are same length before zipping [OK]

Common Mistakes:

• Assuming zip pads shorter list automatically
• Thinking zip causes syntax error
• Believing multiplication is required for hybrid scores

5. You want to improve a hybrid search system by weighting semantic similarity twice as much as keyword matching. Which formula correctly applies this?

hard

A. final_score = semantic_score + 2 * keyword_score

B. final_score = 2 * semantic_score + keyword_score

C. final_score = semantic_score * keyword_score * 2

D. final_score = (semantic_score + keyword_score) / 2

Solution

1. Step 1: Identify weighting requirement

   Semantic similarity should count double compared to keyword score.

2. Step 2: Apply weights in formula

   Multiply semantic_score by 2, then add keyword_score.

3. Final Answer:

   final_score = 2 * semantic_score + keyword_score -> Option B

4. Quick Check:

   Semantic weighted double = 2 * semantic + keyword [OK]

Hint: Multiply semantic score by 2 before adding keyword [OK]

Common Mistakes:

• Weighting keyword score instead of semantic
• Multiplying all scores together
• Dividing sum instead of weighting