Hybrid search (keyword + semantic) in LangChain

Introduction

Hybrid search helps find information by combining exact word matches and understanding the meaning behind words. This makes search results more accurate and useful.

When you want to find documents that contain specific keywords and also relate to the meaning of your query.

When searching a large collection of text where exact matches alone miss relevant results.

When building a search feature that needs to understand user intent better than simple keyword search.

When you want to improve search quality by mixing fast keyword filters with smart semantic ranking.

When users expect both precise and context-aware search results in your app.

Syntax

LangChain

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI

# Create embeddings
embeddings = OpenAIEmbeddings()

# Load or create vector store
vectorstore = FAISS.load_local("index_path", embeddings)

# Define keyword search function (example with simple filter)
def keyword_search(docs, keyword):
    return [doc for doc in docs if keyword.lower() in doc.page_content.lower()]

# Combine keyword and semantic search
query = "your search query"
keyword_results = keyword_search(vectorstore.docstore._dict.values(), "keyword")
semantic_results = vectorstore.similarity_search(query, k=5)

# Merge and rank results (example: semantic results first, then keyword)
combined_results = semantic_results + [doc for doc in keyword_results if doc not in semantic_results]

This example shows how to combine keyword filtering with semantic similarity search using LangChain and FAISS.

You can customize keyword search logic and how to merge results based on your needs.

Examples

Find top 3 documents semantically related to "climate change effects".

LangChain

semantic_results = vectorstore.similarity_search("climate change effects", k=3)

Filter documents containing the word "energy" (case-insensitive).

LangChain

keyword_results = [doc for doc in docs if "energy" in doc.page_content.lower()]

Combine semantic and keyword results, avoiding duplicates.

LangChain

combined_results = semantic_results + [doc for doc in keyword_results if doc not in semantic_results]

Sample Program

This program creates a small set of documents, builds semantic embeddings, and performs both semantic and keyword searches. It then combines the results and prints them.

LangChain

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.schema import Document

# Sample documents
docs = [
    Document(page_content="Climate change impacts on polar bears."),
    Document(page_content="Renewable energy sources and benefits."),
    Document(page_content="Effects of global warming on oceans."),
    Document(page_content="Energy consumption trends in 2023."),
    Document(page_content="Polar bear habitats and climate."),
]

# Create embeddings
embeddings = OpenAIEmbeddings()

# Create vector store from documents
vectorstore = FAISS.from_documents(docs, embeddings)

# Define keyword search function
keyword = "energy"
def keyword_search(documents, keyword):
    return [doc for doc in documents if keyword.lower() in doc.page_content.lower()]

# Perform semantic search
query = "climate change"
semantic_results = vectorstore.similarity_search(query, k=3)

# Perform keyword search
keyword_results = keyword_search(docs, keyword)

# Combine results without duplicates
combined_results = semantic_results + [doc for doc in keyword_results if doc not in semantic_results]

# Print combined results
for i, doc in enumerate(combined_results, 1):
    print(f"Result {i}: {doc.page_content}")

Important Notes

Hybrid search improves search quality by balancing exact matches and meaning.

Keyword search is fast but can miss related ideas; semantic search understands meaning but may be slower.

Adjust how you combine results to fit your app's needs and user expectations.

Summary

Hybrid search mixes keyword and semantic search for better results.

Use keyword search to find exact words and semantic search to find related meanings.

Combining both helps users find what they want more easily.

Practice

1. What is the main advantage of using hybrid search (keyword + semantic) in Langchain?

easy

A. It combines exact word matching with understanding related meanings for better results.

B. It only uses keyword matching for faster searches.

C. It replaces semantic search completely with keyword search.

D. It ignores the meaning of words and focuses on spelling.

Solution

1. Step 1: Understand keyword search

   Keyword search finds exact matches of words typed by the user.

2. Step 2: Understand semantic search

   Semantic search finds results related in meaning, even if words differ.

3. Final Answer:

   It combines exact word matching with understanding related meanings for better results. -> Option A

4. Quick Check:

   Hybrid search = keyword + semantic [OK]

Hint: Remember hybrid means mixing two methods for best results [OK]

Common Mistakes:

• Thinking hybrid search only uses keywords
• Believing semantic search ignores meaning
• Confusing hybrid search with only semantic search

2. Which of the following is the correct way to initialize a hybrid search in Langchain combining keyword and semantic search?

easy

A. HybridSearch(semantic_searcher, keyword_searcher)

B. HybridSearch(keyword_searcher, semantic_searcher)

C. HybridSearch(keyword_searcher + semantic_searcher)

D. HybridSearch(keyword_searcher * semantic_searcher)

Solution

1. Step 1: Check constructor parameters

   HybridSearch expects two separate searcher objects as arguments.

2. Step 2: Confirm order and syntax

   The correct syntax is HybridSearch(keyword_searcher, semantic_searcher) with comma separation.

3. Final Answer:

   HybridSearch(keyword_searcher, semantic_searcher) -> Option B

4. Quick Check:

   Correct syntax uses comma-separated searchers [OK]

Hint: Use commas to separate searchers, not operators [OK]

Common Mistakes:

• Using + or * instead of comma to combine searchers
• Swapping the order of keyword and semantic searchers
• Passing a single combined object instead of two

3. Given the following code snippet in Langchain, what will be the output type of results?

hybrid_search = HybridSearch(keyword_searcher, semantic_searcher)
results = hybrid_search.search('apple')

medium

A. A single document matching the keyword exactly

B. A boolean indicating if 'apple' exists in the database

C. A list of documents ranked by combined keyword and semantic relevance

D. An error because HybridSearch has no search method

Solution

1. Step 1: Understand HybridSearch.search()

   The search method returns multiple documents ranked by relevance.

2. Step 2: Combine keyword and semantic results

   Hybrid search merges keyword and semantic scores to rank documents.

3. Final Answer:

   A list of documents ranked by combined keyword and semantic relevance -> Option C

4. Quick Check:

   Hybrid search returns ranked document list [OK]

Hint: Hybrid search returns ranked lists, not single items [OK]

Common Mistakes:

• Expecting a single document instead of a list
• Thinking it returns a boolean
• Assuming HybridSearch lacks a search method

4. You wrote this code but get an error:

hybrid_search = HybridSearch(keyword_searcher semantic_searcher)
results = hybrid_search.search('banana')

What is the error and how to fix it?

medium

A. Missing comma between keyword_searcher and semantic_searcher; add a comma.

B. search method does not exist; rename to find.

C. HybridSearch cannot take two arguments; combine them first.

D. The string 'banana' should be a list; change to ['banana'].

Solution

1. Step 1: Identify syntax error in constructor

   There is no comma between keyword_searcher and semantic_searcher, causing a syntax error.

2. Step 2: Fix by adding comma

   Insert a comma to separate the two arguments correctly.

3. Final Answer:

   Missing comma between keyword_searcher and semantic_searcher; add a comma. -> Option A

4. Quick Check:

   Arguments must be comma-separated [OK]

Hint: Check commas between arguments to avoid syntax errors [OK]

Common Mistakes:

• Forgetting commas between function arguments
• Assuming method name is wrong
• Changing input type unnecessarily

5. You want to improve search results by weighting keyword matches twice as much as semantic similarity in Langchain's hybrid search. Which approach correctly applies this weighting?

hard

A. Use only keyword_searcher to get stronger keyword matches.

B. Multiply semantic_searcher results by 2 before combining.

C. Swap keyword_searcher and semantic_searcher order in HybridSearch constructor.

D. Use HybridSearch(keyword_searcher, semantic_searcher, weights=[2,1]) to set weights.

Solution

1. Step 1: Understand weighting in hybrid search

   HybridSearch can accept weights to prioritize one search type over another.

2. Step 2: Apply weights parameter correctly

   Passing weights=[2,1] gives keyword search double importance compared to semantic search.

3. Final Answer:

   Use HybridSearch(keyword_searcher, semantic_searcher, weights=[2,1]) to set weights. -> Option D

4. Quick Check:

   Weights parameter controls search importance [OK]

Hint: Use weights parameter to prioritize keyword or semantic search [OK]

Common Mistakes:

• Trying to multiply results manually before combining
• Swapping searchers without weighting
• Using only one searcher loses hybrid benefits