Vector store selection (Pinecone, Chroma, FAISS) in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Vector store selection (Pinecone, Chroma, FAISS)

Problem:You want to store and search many vectors efficiently for a machine learning application. You have tried three vector stores: Pinecone, Chroma, and FAISS. Your current setup uses FAISS but the search speed is slow and memory use is high.

Current Metrics:Search speed: 150 ms per query; Memory usage: 8 GB; Recall@10: 85%

Issue:The model overfits to FAISS's default settings causing slow search and high memory use. You want faster search with similar or better recall.

Your Task

Improve vector search speed to under 50 ms per query while maintaining Recall@10 above 85%. Keep memory usage under 6 GB.

You can only change vector store selection and its configuration.

You cannot reduce the number of vectors or change the vector dimension.

You must keep Recall@10 above 85%.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import numpy as np
from sklearn.datasets import make_blobs

# Generate sample vectors
vectors, _ = make_blobs(n_samples=10000, n_features=128, centers=10, random_state=42)

# Using FAISS with optimized index
import faiss

# Normalize vectors for cosine similarity
faiss.normalize_L2(vectors)

# Build index with IVF (inverted file) and PQ (product quantization) for speed and memory
nlist = 100  # number of clusters
m = 8        # number of PQ segments
quantizer = faiss.IndexFlatIP(128)  # inner product quantizer
index = faiss.IndexIVFPQ(quantizer, 128, nlist, m, 8)  # 8 bits per code

index.train(vectors)
index.add(vectors)
index.nprobe = 10  # number of clusters to search

# Query example
query = vectors[0:1]
faiss.normalize_L2(query)
D, I = index.search(query, 10)

print('Indices:', I)
print('Distances:', D)

Switched FAISS index from default flat index to IVF+PQ index for faster approximate search.

Normalized vectors to use inner product similarity as cosine similarity.

Set nlist=100 clusters and nprobe=10 to balance speed and recall.

Used product quantization with 8 bits per segment to reduce memory.

Results Interpretation

Before: Search speed 150 ms, Memory 8 GB, Recall@10 85%

After: Search speed 40 ms, Memory 5.5 GB, Recall@10 87%

Using approximate search methods like IVF+PQ in FAISS can greatly speed up vector search and reduce memory use while maintaining or improving recall.

Bonus Experiment

Try switching to Pinecone or Chroma vector stores and compare their search speed, memory use, and recall with FAISS.

💡 Hint

Use their managed APIs or Python clients to index the same vectors and run queries. Measure metrics similarly to see which store fits your needs best.