Why embeddings capture semantic meaning in Prompt Engineering / GenAI - Why Metrics Matter

Metrics & Evaluation - Why embeddings capture semantic meaning

Which metric matters for this concept and WHY

When we talk about embeddings capturing semantic meaning, the key metric is cosine similarity. This metric measures how close two vectors are in direction, regardless of their length. Since embeddings are vectors representing words or sentences, cosine similarity tells us how similar their meanings are. A higher cosine similarity means the embeddings share more semantic meaning.

Confusion matrix or equivalent visualization (ASCII)

Example: Comparing embeddings of words "cat", "dog", and "car" using cosine similarity

          cat     dog     car
cat     1.00    0.85    0.10
dog     0.85    1.00    0.12
car     0.10    0.12    1.00

Here, "cat" and "dog" have high similarity (0.85), showing semantic closeness.
"cat" and "car" have low similarity (0.10), showing different meanings.

Precision vs Recall tradeoff with concrete examples

In semantic search or recommendation systems using embeddings, precision means how many of the retrieved items are truly relevant (semantically close). Recall means how many of all relevant items were found.

For example, if you search for "apple" meaning the fruit, high precision means most results are about fruit, not the company. High recall means you find most fruit-related items.

Sometimes increasing recall (finding more related items) lowers precision (some unrelated items appear). Balancing these depends on the application.

What "good" vs "bad" metric values look like for this use case

A good embedding model will have:

• High cosine similarity (close to 1) for semantically similar words or sentences.
• Low cosine similarity (close to 0 or negative) for unrelated meanings.

A bad model might show high similarity for unrelated words, confusing meanings, or low similarity for synonyms.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Ignoring vector length: Using Euclidean distance instead of cosine similarity can mislead semantic closeness.
• Overfitting embeddings: Embeddings trained on small data may memorize instead of generalizing meaning.
• Data leakage: If test words appear in training, similarity scores may be artificially high.
• Ignoring context: Static embeddings ignore word meaning changes in sentences, lowering real semantic capture.

Self-check question

Your embedding model shows cosine similarity of 0.95 between "bank" (financial) and "river". Is this good? Why or why not?

Answer: No, this is not good. "Bank" and "river" have different meanings here. High similarity means the model confuses meanings and does not capture semantic differences well.

Key Result

Cosine similarity is the key metric showing how well embeddings capture semantic meaning by measuring vector closeness.

Practice

1. Why do embeddings help computers understand language better?

easy

A. Because they store words as images

B. Because they turn words into numbers that show meaning

C. Because they translate words into different languages

D. Because they count how many letters are in a word

Solution

1. Step 1: Understand what embeddings do

   Embeddings convert words or ideas into numbers that capture their meaning.

2. Step 2: Recognize why this helps computers

   Numbers allow computers to compare and find similarities between words easily.

3. Final Answer:

   Because they turn words into numbers that show meaning -> Option B

4. Quick Check:

   Embeddings = numbers showing meaning [OK]

Hint: Embeddings = numbers that capture meaning [OK]

Common Mistakes:

• Thinking embeddings store images
• Confusing embeddings with translation
• Believing embeddings count letters

2. Which of the following is the correct way to say embeddings capture semantic meaning?

easy

A. Embeddings count the frequency of words

B. Embeddings store words as raw text strings

C. Embeddings translate words into pictures

D. Embeddings map words to vectors of numbers

Solution

1. Step 1: Identify the correct technical description

   Embeddings represent words as vectors (lists) of numbers.

2. Step 2: Eliminate incorrect options

   Raw text, pictures, and frequency counts do not capture semantic meaning as embeddings do.

3. Final Answer:

   Embeddings map words to vectors of numbers -> Option D

4. Quick Check:

   Embeddings = vectors of numbers [OK]

Hint: Embeddings = vectors, not raw text or images [OK]

Common Mistakes:

• Confusing embeddings with raw text storage
• Thinking embeddings are images
• Mixing embeddings with word counts

3. Given two embeddings: embedding1 = [0.1, 0.3, 0.5] and embedding2 = [0.1, 0.31, 0.49], what can we say about their semantic similarity?

medium

A. They have no relation in meaning

B. They are very different in meaning

C. They are somewhat similar in meaning

D. They are exactly the same meaning

Solution

1. Step 1: Compare the two embeddings numerically

   The numbers are close but not identical, showing some similarity.

2. Step 2: Understand what closeness means in embeddings

   Close embeddings mean similar meanings, but not exactly the same.

3. Final Answer:

   They are somewhat similar in meaning -> Option C

4. Quick Check:

   Close vectors = similar meaning [OK]

Hint: Close embeddings mean similar meaning [OK]

Common Mistakes:

• Assuming small differences mean no similarity
• Thinking embeddings must be identical to be similar
• Ignoring numerical closeness

4. Look at this code snippet that tries to find similarity between two embeddings:

embedding1 = [0.2, 0.4, 0.6]
embedding2 = [0.2, 0.4, 0.6]

similarity = sum(embedding1[i] * embedding2[i] for i in range(3))
print(similarity)

What is the error in this code?

medium

A. The code correctly computes dot product similarity

B. The code should normalize embeddings before dot product

C. The code uses sum incorrectly; it should use a loop

D. The code uses wrong indices for embeddings

Solution

1. Step 1: Analyze the code logic

   The code calculates the dot product by summing element-wise products.

2. Step 2: Check if this is a valid similarity measure

   Dot product is a common way to measure similarity between embeddings.

3. Final Answer:

   The code correctly computes dot product similarity -> Option A

4. Quick Check:

   Dot product code is correct [OK]

Hint: Dot product sums element-wise products [OK]

Common Mistakes:

• Thinking sum can't be used with generator expressions
• Believing normalization is always required
• Confusing indices usage

5. You have embeddings for words: 'cat', 'dog', and 'car'. Which embedding pair is expected to be closest in meaning and why?

hard

A. Embeddings of 'cat' and 'dog' because both are animals

B. Embeddings of 'cat' and 'car' because they start with the same letter

C. Embeddings of 'dog' and 'car' because they have the same number of letters

D. Embeddings of 'cat' and 'dog' because they rhyme

Solution

1. Step 1: Understand semantic meaning in embeddings

   Embeddings capture meaning, so similar concepts have closer embeddings.

2. Step 2: Compare the word pairs by meaning

   'Cat' and 'dog' are both animals, so their embeddings should be closer than unrelated words.

3. Final Answer:

   Embeddings of 'cat' and 'dog' because both are animals -> Option A

4. Quick Check:

   Similar meaning = closer embeddings [OK]

Hint: Semantic similarity beats spelling or sound [OK]

Common Mistakes:

• Choosing words based on spelling or sound
• Ignoring actual meaning of words
• Assuming letter count affects embeddings