OpenAI embeddings API in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - OpenAI embeddings API

Which metric matters for OpenAI embeddings API and WHY

When using OpenAI embeddings, the key metric is cosine similarity. This measures how close two vectors are in direction, showing how similar two pieces of text are. A higher cosine similarity means the texts are more alike. This metric matters because embeddings turn words or sentences into numbers, and cosine similarity helps find related meanings or topics.

Confusion matrix or equivalent visualization

Embeddings do not use a confusion matrix because they are not classification models. Instead, similarity scores between vectors are used. For example, if you embed two sentences and get a cosine similarity of 0.9, they are very similar. If the score is 0.1, they are very different.

Example cosine similarity scores:
Sentence A & Sentence B: 0.92 (very similar)
Sentence A & Sentence C: 0.15 (not similar)
Sentence B & Sentence C: 0.20 (not similar)
    

Precision vs Recall tradeoff with concrete examples

In embedding search, precision means how many of the retrieved items are truly relevant, while recall means how many relevant items were found out of all possible relevant items.

For example, if you search for documents similar to a query, high precision means the top results are very relevant. High recall means you find most of the relevant documents, even if some less relevant ones appear.

Tradeoff:

• High precision, low recall: You get very accurate results but might miss some relevant ones.
• High recall, low precision: You find most relevant results but also get many irrelevant ones.

Depending on the use case, you might prefer one over the other. For example, a legal document search needs high recall to not miss any important case, while a recommendation system might prefer high precision to show only the best matches.

What "good" vs "bad" metric values look like for OpenAI embeddings

Good values:

• Cosine similarity close to 1.0 for truly similar texts.
• High precision and recall in retrieval tasks (e.g., precision > 0.8, recall > 0.7).
• Consistent similarity scores that match human judgment.

Bad values:

• Cosine similarity near 0 or negative for similar texts (shows embeddings are not capturing meaning).
• Low precision or recall, meaning many irrelevant results or many relevant results missed.
• Unstable or inconsistent similarity scores across similar inputs.

Metrics pitfalls

• Using accuracy: Accuracy is not meaningful because embeddings are not classifiers.
• Ignoring context: Embeddings depend on context; comparing unrelated texts can give misleading similarity.
• Data leakage: Using test data in training embeddings can inflate similarity scores.
• Overfitting: Over-specialized embeddings may not generalize well to new texts.
• Threshold choice: Picking a similarity threshold without validation can cause poor retrieval results.

Self-check question

Your embedding-based search returns results with an average cosine similarity of 0.98 for relevant documents but only 0.3 for some documents you know are related. Is your model good? Why or why not?

Answer: The model is good at finding very similar documents (0.98 similarity) but misses some related ones (0.3 similarity). This means it has high precision but low recall. Depending on your goal, you might want to improve recall by adjusting thresholds or using better embeddings.

Key Result

Cosine similarity is the key metric for OpenAI embeddings, measuring how close two texts are in meaning.