Vector databases (Pinecone, ChromaDB, Weaviate) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Vector databases (Pinecone, ChromaDB, Weaviate)

Which metric matters for Vector Databases and WHY

Vector databases store and search data by similarity, not exact matches. The key metric is Recall, which tells us how many of the truly similar items the database finds. High recall means the database finds most relevant vectors, important for good search results.

Precision also matters because it shows how many found items are actually relevant. But recall is often more critical because missing relevant items hurts user experience more than extra irrelevant ones.

Another important metric is Latency -- how fast the database returns results. Fast responses keep users happy.

Confusion Matrix for Vector Search

    |---------------------------|
    |           | Predicted     |
    | Actual    | Similar | Not |
    |-----------|---------|-----|
    | Similar   | TP      | FN  |
    | Not       | FP      | TN  |
    |---------------------------|

    TP = Relevant vectors found
    FN = Relevant vectors missed
    FP = Irrelevant vectors found
    TN = Irrelevant vectors not found
    

Recall = TP / (TP + FN) shows how many relevant vectors were found.

Precision = TP / (TP + FP) shows how many found vectors are relevant.

Precision vs Recall Tradeoff with Examples

Imagine a vector database for a movie recommendation app.

• High Recall, Lower Precision: The database returns many movies similar to your favorite, including some less relevant ones. You get more options but some may not fit your taste.
• High Precision, Lower Recall: The database returns only very close matches. You get fewer options but they are very relevant.

For discovery, high recall is better to not miss good options. For strict filtering, high precision is better to avoid irrelevant results.

What Good vs Bad Metrics Look Like for Vector Databases

• Good: Recall above 90%, Precision above 80%, Latency under 100ms. Most relevant vectors found quickly.
• Bad: Recall below 50%, Precision below 50%, Latency over 500ms. Many relevant vectors missed or slow responses.

Good metrics mean users find what they want fast. Bad metrics mean poor search experience.

Common Pitfalls in Vector Database Metrics

• Ignoring Recall: Focusing only on precision can miss many relevant vectors.
• Data Leakage: Testing on vectors already in the database inflates metrics.
• Overfitting: Tuning only for a small test set can hurt real-world performance.
• Latency Overlooked: Fast search is critical; slow queries frustrate users.

Self Check

Your vector database returns results with 98% precision but only 12% recall. Is it good for production?

Answer: No. While most returned vectors are relevant (high precision), the database misses most relevant vectors (very low recall). Users will not see many good matches, hurting experience.

Key Result

Recall is key for vector databases to find most relevant items; precision and latency also matter for quality and speed.