OpenAI embeddings API in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - OpenAI embeddings API

Which metric matters for OpenAI embeddings API and WHY

When using OpenAI embeddings, the key metric is cosine similarity. This measures how close two vectors are in direction, showing how similar two pieces of text are. A higher cosine similarity means the texts are more alike. This metric matters because embeddings turn words or sentences into numbers, and cosine similarity helps find related meanings or topics.

Confusion matrix or equivalent visualization

Embeddings do not use a confusion matrix because they are not classification models. Instead, similarity scores between vectors are used. For example, if you embed two sentences and get a cosine similarity of 0.9, they are very similar. If the score is 0.1, they are very different.

Example cosine similarity scores:
Sentence A & Sentence B: 0.92 (very similar)
Sentence A & Sentence C: 0.15 (not similar)
Sentence B & Sentence C: 0.20 (not similar)
    

Precision vs Recall tradeoff with concrete examples

In embedding search, precision means how many of the retrieved items are truly relevant, while recall means how many relevant items were found out of all possible relevant items.

For example, if you search for documents similar to a query, high precision means the top results are very relevant. High recall means you find most of the relevant documents, even if some less relevant ones appear.

Tradeoff:

• High precision, low recall: You get very accurate results but might miss some relevant ones.
• High recall, low precision: You find most relevant results but also get many irrelevant ones.

Depending on the use case, you might prefer one over the other. For example, a legal document search needs high recall to not miss any important case, while a recommendation system might prefer high precision to show only the best matches.

What "good" vs "bad" metric values look like for OpenAI embeddings

Good values:

• Cosine similarity close to 1.0 for truly similar texts.
• High precision and recall in retrieval tasks (e.g., precision > 0.8, recall > 0.7).
• Consistent similarity scores that match human judgment.

Bad values:

• Cosine similarity near 0 or negative for similar texts (shows embeddings are not capturing meaning).
• Low precision or recall, meaning many irrelevant results or many relevant results missed.
• Unstable or inconsistent similarity scores across similar inputs.

Metrics pitfalls

• Using accuracy: Accuracy is not meaningful because embeddings are not classifiers.
• Ignoring context: Embeddings depend on context; comparing unrelated texts can give misleading similarity.
• Data leakage: Using test data in training embeddings can inflate similarity scores.
• Overfitting: Over-specialized embeddings may not generalize well to new texts.
• Threshold choice: Picking a similarity threshold without validation can cause poor retrieval results.

Self-check question

Your embedding-based search returns results with an average cosine similarity of 0.98 for relevant documents but only 0.3 for some documents you know are related. Is your model good? Why or why not?

Answer: The model is good at finding very similar documents (0.98 similarity) but misses some related ones (0.3 similarity). This means it has high precision but low recall. Depending on your goal, you might want to improve recall by adjusting thresholds or using better embeddings.

Key Result

Cosine similarity is the key metric for OpenAI embeddings, measuring how close two texts are in meaning.

Practice

1. What does the OpenAI embeddings API primarily do?

easy

A. Translates text from one language to another

B. Generates images from text descriptions

C. Converts text into number vectors to capture meaning

D. Summarizes long documents into short paragraphs

Solution

1. Step 1: Understand the purpose of embeddings

   Embeddings are numeric representations of text that capture its meaning.

2. Step 2: Match the API function

   The OpenAI embeddings API converts text into these numeric vectors.

3. Final Answer:

   Converts text into number vectors to capture meaning -> Option C

4. Quick Check:

   Embeddings = numeric text vectors [OK]

Hint: Embeddings turn words into numbers for computers [OK]

Common Mistakes:

• Confusing embeddings with image generation
• Thinking embeddings translate languages
• Assuming embeddings summarize text

2. Which of the following is the correct way to call the OpenAI embeddings API in Python?

easy

A. openai.Embeddings.generate(text='text', model='embedding-3')

B. openai.Embedding.create(input=['text'], model='text-embedding-3-large')

C. openai.embedding.create(text='text', model='text-embedding-3-large')

D. openai.Embedding.create(input='text', model='text-embedding-3-small')

Solution

1. Step 1: Recall correct method and parameters

   The correct method is openai.Embedding.create with 'input' as a list of texts and a valid model name.

2. Step 2: Check each option

   openai.Embedding.create(input=['text'], model='text-embedding-3-large') uses correct method, parameter name 'input' as a list, and a valid model name.

3. Final Answer:

   openai.Embedding.create(input=['text'], model='text-embedding-3-large') -> Option B

4. Quick Check:

   Correct method and input list = A [OK]

Hint: Use 'Embedding.create' with input list and model name [OK]

Common Mistakes:

• Using wrong method name like Embeddings.generate
• Passing input as string instead of list
• Incorrect parameter names like 'text' instead of 'input'

3. What will be the output type of the following Python code snippet using OpenAI embeddings API?

response = openai.Embedding.create(input=['hello world'], model='text-embedding-3-large')
embedding_vector = response['data'][0]['embedding']
print(type(embedding_vector))

medium

A. <class 'list'>

B. <class 'dict'>

C. <class 'float'>

D. <class 'str'>

Solution

1. Step 1: Understand the API response structure

   The 'embedding' field contains a list of floats representing the vector.

2. Step 2: Check the type of 'embedding_vector'

   Extracting response['data'][0]['embedding'] returns a list of numbers.

3. Final Answer:

   <class 'list'> -> Option A

4. Quick Check:

   Embedding vector is a list of floats [OK]

Hint: Embedding is a list of numbers, not a single value [OK]

Common Mistakes:

• Assuming embedding is a dict or string
• Thinking embedding is a single float
• Confusing API response with raw text

4. Identify the error in this code snippet using OpenAI embeddings API:

response = openai.Embedding.create(input='hello world', model='text-embedding-3-large')
embedding = response['data'][0]['embedding']
print(len(embedding))

medium

A. The print statement should be print(embedding.length)

B. The model name 'text-embedding-3-large' is invalid

C. The 'embedding' key does not exist in the response

D. The 'input' parameter should be a list, not a string

Solution

1. Step 1: Check the 'input' parameter type

   The API expects 'input' as a list of strings, not a single string.

2. Step 2: Identify the error cause

   Passing a string causes the API to error or behave unexpectedly.

3. Final Answer:

   The 'input' parameter should be a list, not a string -> Option D

4. Quick Check:

   Input must be list, not string [OK]

Hint: Always pass input as a list of texts [OK]

Common Mistakes:

• Passing input as a single string
• Using wrong model names
• Incorrect print syntax for length

5. You want to find the similarity between two sentences using OpenAI embeddings API. Which approach is correct?

hard

A. Get embeddings for both sentences, then compute cosine similarity between vectors

B. Send both sentences as one string to embeddings API and compare output length

C. Use embeddings API to translate sentences, then compare translated texts

D. Get embeddings for one sentence only and compare with raw text of the other

Solution

1. Step 1: Understand similarity calculation with embeddings

   Similarity is measured by comparing numeric vectors, usually with cosine similarity.

2. Step 2: Apply correct method

   Get embeddings separately for each sentence, then compute cosine similarity between their vectors.

3. Final Answer:

   Get embeddings for both sentences, then compute cosine similarity between vectors -> Option A

4. Quick Check:

   Similarity = cosine of embedding vectors [OK]

Hint: Compare vectors with cosine similarity after embedding [OK]

Common Mistakes:

• Combining sentences into one string before embedding
• Comparing raw text lengths instead of vectors
• Using embeddings for only one sentence