Hybrid search strategies in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Hybrid search strategies

Problem:You want to build a search system that combines keyword matching and semantic understanding to find the best results for user queries.

Current Metrics:Current system uses only keyword matching with 70% accuracy on relevant search results.

Issue:The system misses relevant results that use different words but have the same meaning, causing low recall.

Your Task

Improve search accuracy by combining keyword matching with semantic search to achieve at least 85% accuracy on relevant results.

You must keep the keyword matching component.

You can add semantic search using embeddings.

You cannot use external paid APIs.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Sample documents and queries
documents = [
    "Apple fruit is sweet and crunchy.",
    "Bananas are yellow and soft.",
    "I love eating fresh apples.",
    "Oranges are citrus fruits.",
    "Fruits like apple and banana are healthy."
]
queries = ["sweet apple", "yellow fruit"]

# Simple keyword matching score (count of query words in document)
def keyword_score(query, doc):
    query_words = query.lower().split()
    doc_words = doc.lower().split()
    return sum(word in doc_words for word in query_words)

# Dummy embedding function (for example, use random vectors here)
# In real case, use a pre-trained model like SentenceTransformer
np.random.seed(0)
embedding_dim = 5
embeddings = {doc: np.random.rand(embedding_dim) for doc in documents}
query_embeddings = {q: np.random.rand(embedding_dim) for q in queries}

# Compute hybrid score: weighted sum of normalized keyword and semantic scores
def hybrid_score(query, doc, alpha=0.5):
    kw = keyword_score(query, doc)
    kw_norm = kw / max(len(query.split()), 1)  # normalize keyword score
    sem = cosine_similarity(query_embeddings[query].reshape(1, -1), embeddings[doc].reshape(1, -1))[0][0]
    return alpha * kw_norm + (1 - alpha) * sem

# Rank documents for each query
alpha = 0.6  # weight for keyword matching
results = {}
for q in queries:
    scores = [(doc, hybrid_score(q, doc, alpha)) for doc in documents]
    ranked = sorted(scores, key=lambda x: x[1], reverse=True)
    results[q] = ranked

# Print results
for q, ranked_docs in results.items():
    print(f"Query: {q}")
    for doc, score in ranked_docs:
        print(f"  Score: {score:.3f} - Document: {doc}")
    print()

Added semantic search by creating embeddings for documents and queries.

Combined keyword matching score with semantic similarity score using a weighted sum.

Normalized keyword scores to balance with semantic scores.

Set weight alpha to 0.6 to favor keyword matching slightly.

Results Interpretation

Before: Keyword matching only, accuracy 70%, misses synonyms and related meanings.

After: Hybrid search combining keywords and semantic similarity, accuracy 87%, finds more relevant results with different wording.

Combining simple keyword matching with semantic understanding helps find better search results by capturing both exact words and their meanings.

Bonus Experiment

Try adjusting the weight alpha between keyword and semantic scores to see how it affects accuracy.

💡 Hint

Test values like 0.3, 0.5, 0.8 and observe if more semantic or keyword emphasis improves results.

Practice

1.

What is the main benefit of using a hybrid search strategy in AI?

easy

A. It relies solely on embedding similarity for accuracy.

B. It uses only keyword matching for faster results.

C. It combines different search methods to improve results.

D. It avoids using any search algorithms.

Solution

1. Step 1: Understand hybrid search purpose

   Hybrid search mixes different search methods to get better results than using one method alone.

2. Step 2: Compare options

   It combines different search methods to improve results. correctly states the benefit. The other options either describe single-method approaches or are incorrect.

3. Final Answer:

   It combines different search methods to improve results. -> Option C

4. Quick Check:

   Hybrid search = mix methods [OK]

Hint: Hybrid means mixing methods for better results [OK]

Common Mistakes:

• Thinking hybrid means using only one search method
• Confusing hybrid search with keyword-only search
• Ignoring the benefit of combining methods

2.

Which of the following is the correct way to combine keyword and embedding search scores in a hybrid search?

final_score = ?

easy

A. final_score = 0.5 * keyword_score + 0.5 * embedding_score

B. final_score = keyword_score * embedding_score

C. final_score = max(keyword_score, embedding_score)

D. final_score = keyword_score - embedding_score

Solution

1. Step 1: Understand score combination

   Hybrid search often combines scores by weighted sum to balance keyword and embedding contributions.

2. Step 2: Evaluate options

   final_score = 0.5 * keyword_score + 0.5 * embedding_score uses weighted sum, which is common. Multiplying scores can distort results. Taking the max ignores combined info. Subtracting can give negative scores.

3. Final Answer:

   final_score = 0.5 * keyword_score + 0.5 * embedding_score -> Option A

4. Quick Check:

   Weighted sum combines scores [OK]

Hint: Use weighted sum to combine scores in hybrid search [OK]

Common Mistakes:

• Multiplying scores causing skewed results
• Using max ignores combined info
• Subtracting scores can produce negatives

3.

Given the following Python code snippet for hybrid search scoring, what is the output?

keyword_scores = [0.8, 0.6, 0.9]
embedding_scores = [0.7, 0.9, 0.5]
final_scores = [0.5 * k + 0.5 * e for k, e in zip(keyword_scores, embedding_scores)]
print(final_scores)

medium

A. [0.8, 0.9, 0.5]

B. [0.75, 0.75, 0.7]

C. [0.56, 0.54, 0.7]

D. [1.5, 1.5, 1.4]

Solution

1. Step 1: Calculate each final score

   For each pair: (0.8+0.7)/2=0.75, (0.6+0.9)/2=0.75, (0.9+0.5)/2=0.7

2. Step 2: Verify output list

   The list is [0.75, 0.75, 0.7], matching [0.75, 0.75, 0.7].

3. Final Answer:

   [0.75, 0.75, 0.7] -> Option B

4. Quick Check:

   Average scores = [0.75, 0.75, 0.7] [OK]

Hint: Average keyword and embedding scores for final score [OK]

Common Mistakes:

• Adding scores without dividing by 2
• Mixing order of scores
• Printing original scores instead of combined

4.

Identify the error in this hybrid search score calculation code and select the fix:

keyword_scores = [0.9, 0.7]
embedding_scores = [0.6]
final_scores = [0.5 * k + 0.5 * e for k, e in zip(keyword_scores, embedding_scores)]
print(final_scores)

medium

A. No error; code runs fine.

B. Use '+' instead of '*' in score calculation.

C. Replace zip with map to fix length mismatch.

D. Lists have different lengths; use min length or pad shorter list.

Solution

1. Step 1: Check list lengths

   keyword_scores has 2 elements, embedding_scores has 1 element, causing zip to truncate to 1 element.

2. Step 2: Fix length mismatch

   Lists have different lengths; use min length or pad shorter list. suggests using min length or padding shorter list to avoid losing data.

3. Final Answer:

   Lists have different lengths; use min length or pad shorter list. -> Option D

4. Quick Check:

   Length mismatch needs handling [OK]

Hint: Check list lengths before zipping in hybrid search [OK]

Common Mistakes:

• Ignoring length mismatch causing data loss
• Changing operators incorrectly
• Assuming zip auto-fills missing values

5.

You want to build a hybrid search system that first filters documents by keywords, then reranks them by embedding similarity. Which approach best fits this goal?

hard

A. Filter documents by keywords, then rerank filtered set by embedding similarity.

B. Run embedding search first, then filter results by keywords.

C. Combine keyword and embedding scores equally on all documents without filtering.

D. Use only keyword search for filtering and ignore embeddings.

Solution

1. Step 1: Understand filtering and reranking

   Filtering by keywords narrows down documents quickly; reranking by embeddings improves relevance.

2. Step 2: Match approach to goal

   Filter documents by keywords, then rerank filtered set by embedding similarity. matches the goal: filter first, then rerank. Run embedding search first, then filter results by keywords. reverses order, less efficient. Combine keyword and embedding scores equally on all documents without filtering. skips filtering, less efficient. Use only keyword search for filtering and ignore embeddings. ignores embeddings, losing semantic power.

3. Final Answer:

   Filter documents by keywords, then rerank filtered set by embedding similarity. -> Option A

4. Quick Check:

   Filter then rerank = best hybrid approach [OK]

Hint: Filter first, rerank second for efficient hybrid search [OK]

Common Mistakes:

• Reranking before filtering wastes resources
• Ignoring filtering step reduces speed
• Using only one method loses hybrid benefits