Metadata filtering in vector stores in LangChain

Introduction

Metadata filtering helps you find specific items in a vector store by using extra information about those items. It makes searching faster and more accurate.

You want to search documents but only from a certain date or author.

You have many items and want to narrow results by category or tag.

You want to exclude some items from search results based on their properties.

You want to combine text similarity with specific conditions like language or type.

Syntax

LangChain

results = vector_store.similarity_search(query, filter={"key": "value"})

The filter is a dictionary with metadata keys and values to match.

Only items with metadata matching the filter are returned.

Examples

Search for items about 'climate change' but only from the year 2023.

LangChain

results = vector_store.similarity_search("climate change", filter={"year": "2023"})

Find dessert recipes that are easy to make.

LangChain

results = vector_store.similarity_search("recipe", filter={"category": "dessert", "difficulty": "easy"})

Search machine learning documents only in English.

LangChain

results = vector_store.similarity_search("machine learning", filter={"language": "English"})

Sample Program

This example creates a vector store with three documents, each having metadata about topic and level. It searches for documents related to 'Python' but only those tagged as programming and beginner level. The output shows matching document texts.

LangChain

from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings

# Sample documents with metadata
documents = [
    {"text": "Learn Python basics.", "metadata": {"topic": "programming", "level": "beginner"}},
    {"text": "Advanced Python techniques.", "metadata": {"topic": "programming", "level": "advanced"}},
    {"text": "Cooking pasta recipes.", "metadata": {"topic": "cooking", "level": "beginner"}}
]

# Create embeddings
embeddings = OpenAIEmbeddings()

# Prepare texts and metadata
texts = [doc["text"] for doc in documents]
metadatas = [doc["metadata"] for doc in documents]

# Create vector store with metadata
vector_store = FAISS.from_texts(texts, embeddings, metadatas=metadatas)

# Search for programming documents at beginner level
results = vector_store.similarity_search("Python", filter={"topic": "programming", "level": "beginner"})

for r in results:
    print(r.page_content)

Important Notes

Filters must match metadata keys exactly as stored.

Not all vector stores support metadata filtering; check your store's documentation.

Filtering helps reduce noise and improves search relevance.

Summary

Metadata filtering narrows search results by extra information.

Use a dictionary with key-value pairs to filter results.

It works well when you want specific categories or properties in your search.

Practice

1. What is the main purpose of metadata filtering in vector stores?

easy

A. To delete irrelevant data from the vector store

B. To increase the size of the vector store

C. To speed up vector calculations without filtering

D. To narrow search results using extra information about data

Solution

1. Step 1: Understand metadata filtering concept

   Metadata filtering uses extra information (metadata) to limit search results to relevant items.

2. Step 2: Identify the main purpose

   The main goal is to narrow down results, not to delete data or speed up calculations without filtering.

3. Final Answer:

   To narrow search results using extra information about data -> Option D

4. Quick Check:

   Metadata filtering = Narrow search results [OK]

Hint: Metadata filters results by extra info, not by size or speed [OK]

Common Mistakes:

• Confusing filtering with deleting data
• Thinking metadata speeds up vector math alone
• Believing filtering increases store size

2. Which of the following is the correct way to apply metadata filtering in a Langchain vector store search?

easy

A. search(query, filter=[('category', 'news')])

B. search(query, filter='category=news')

C. search(query, filter={'category': 'news'})

D. search(query, filter='category:news')

Solution

1. Step 1: Recall correct filter syntax

   Langchain expects a dictionary for metadata filtering, e.g., {'key': 'value'}.

2. Step 2: Match syntax to options

   Only search(query, filter={'category': 'news'}) uses a dictionary correctly; others use strings or lists incorrectly.

3. Final Answer:

   search(query, filter={'category': 'news'}) -> Option C

4. Quick Check:

   Filter syntax = dictionary format [OK]

Hint: Use dictionary for filters: filter={'key': 'value'} [OK]

Common Mistakes:

• Using string instead of dictionary for filter
• Passing list of tuples instead of dict
• Using colon syntax inside string

3. Given this code snippet using Langchain vector store:

results = vector_store.search('climate change', filter={'source': 'news', 'year': 2023})
print(len(results))

If the vector store has 10 documents with source 'news' and year 2023, and 5 with source 'news' but year 2022, what will be printed?

medium

A. 15

B. 10

C. 5

D. 0

Solution

1. Step 1: Understand filter effect

   The filter requires both 'source' = 'news' and 'year' = 2023 to match.

2. Step 2: Count matching documents

   Only 10 documents match both conditions; 5 with year 2022 do not match.

3. Final Answer:

   10 -> Option B

4. Quick Check:

   Filter matches 10 docs with both conditions [OK]

Hint: Filter matches all conditions exactly, not partial [OK]

Common Mistakes:

• Counting documents matching only one filter key
• Ignoring year condition
• Assuming filter returns all 'news' documents

4. You wrote this code but get an error:

results = vector_store.search('AI', filter=['category', 'tech'])

What is the main problem causing the error?

medium

A. Filter should be a dictionary, not a list

B. Search query must be a dictionary

C. Filter keys must be integers

D. Filter cannot be used with string queries

Solution

1. Step 1: Check filter type

   The filter parameter expects a dictionary with key-value pairs.

2. Step 2: Identify incorrect usage

   The code uses a list instead of a dictionary, causing a type error.

3. Final Answer:

   Filter should be a dictionary, not a list -> Option A

4. Quick Check:

   Filter type must be dict, not list [OK]

Hint: Always use dict for filter, not list or other types [OK]

Common Mistakes:

• Passing list instead of dict for filter
• Thinking query must be dict
• Believing filter keys must be numbers

5. You want to find documents in a Langchain vector store about 'machine learning' but only from the year 2022 or later, and with the tag 'tutorial'. Which filter dictionary correctly applies this condition?

hard

A. {'tag': 'tutorial', 'year': {'$gte': 2022}}

B. {'tag': 'tutorial', 'year': 2022}

C. {'tag': ['tutorial'], 'year': 2022}

D. {'tag': 'tutorial', 'year': [2022, 2023, 2024]}

Solution

1. Step 1: Understand filtering for range conditions

   To filter for year 2022 or later, a comparison operator like '$gte' (greater than or equal) is needed.

2. Step 2: Analyze options for correct syntax

   {'tag': 'tutorial', 'year': {'$gte': 2022}} uses {'year': {'$gte': 2022}} which correctly expresses the condition; others do not support range filtering properly.

3. Final Answer:

   {'tag': 'tutorial', 'year': {'$gte': 2022}} -> Option A

4. Quick Check:

   Use comparison operators for range filters [OK]

Hint: Use {'key': {'$gte': value}} for filtering ranges [OK]

Common Mistakes:

• Using exact value instead of range operator
• Passing list instead of operator for year
• Misunderstanding how to filter numeric ranges