Text embedding models in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Text embedding models

Which metric matters for Text embedding models and WHY

Text embedding models turn words or sentences into numbers so computers can understand them. To check how good these numbers are, we use cosine similarity or distance metrics. These tell us if similar texts have close embeddings and different texts are far apart. For tasks like search or recommendation, precision@k and recall@k show how well the model finds relevant items among top results.

Confusion matrix or equivalent visualization

Text embedding models usually don't use confusion matrices directly because they output vectors, not class labels. Instead, we look at similarity scores. Here is a simple example of similarity scores for 3 pairs:

    Pair             | Similarity Score
    -----------------|-----------------
    "cat" vs "dog" | 0.85 (high, related)
    "cat" vs "car" | 0.30 (low, unrelated)
    "dog" vs "wolf"| 0.90 (very high, related)
    

High scores mean embeddings are close, showing the model understands meaning well.

Precision vs Recall tradeoff with concrete examples

Imagine a search engine using embeddings. If it shows only very few results (high precision), it might miss some good answers (low recall). If it shows many results (high recall), some might be less relevant (low precision). For example:

• High precision, low recall: Only top 3 very close matches shown, but misses other good ones.
• High recall, low precision: Shows 20 results including many not related.

Balancing precision and recall depends on what users want: very accurate few results or more complete but less precise results.

What "good" vs "bad" metric values look like for Text embedding models

Good embedding models have:

• High cosine similarity (close to 1.0) for related texts.
• Low cosine similarity (close to 0 or negative) for unrelated texts.
• Precision@10 above 0.7 means most top 10 results are relevant.
• Recall@10 above 0.6 means it finds most relevant items in top 10.

Bad models show similar scores for unrelated texts or low precision and recall, meaning embeddings do not capture meaning well.

Metrics pitfalls

• Using accuracy: Accuracy is not useful because embeddings are vectors, not classes.
• Ignoring data diversity: Testing only on similar texts can hide poor performance on different topics.
• Overfitting: Model may memorize training pairs, showing high similarity only on known data.
• Data leakage: If test texts appear in training, metrics look better but model is not truly generalizing.
• Ignoring metric choice: Using Euclidean distance instead of cosine similarity can give misleading results.

Self-check question

Your text embedding model shows cosine similarity 0.95 for unrelated texts and 0.60 for related texts. Is it good? Why or why not?

Answer: No, it is not good. Related texts should have higher similarity than unrelated ones. Here, unrelated texts have higher similarity (0.95) than related (0.60), so the model fails to capture meaning properly.

Key Result

Cosine similarity and precision@k are key metrics to evaluate how well text embeddings capture meaning and relevance.