Word embeddings concept (Word2Vec) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Word embeddings concept (Word2Vec)

Which metric matters for Word2Vec and WHY

Word2Vec creates word vectors that capture meaning. To check if it works well, we use cosine similarity. This measures how close two word vectors are in meaning. A higher cosine similarity means words are used in similar ways.

We also look at analogy accuracy. For example, if the model can solve "king - man + woman = ?" and find "queen", it shows the embeddings learned meaningful relationships.

Confusion matrix or equivalent visualization

Word2Vec does not use a confusion matrix like classification. Instead, we visualize word vectors in 2D using t-SNE or PCA. Words with similar meanings cluster together.

    Example 2D plot:
    
    [king]      [queen]
       \        /
        \      /
         [royal]
          |
    [man]  [woman]
    
    Words close together show the model learned meaning well.
    

Precision vs Recall tradeoff with examples

In Word2Vec, precision and recall are less direct but relate to how well the model finds similar words.

• High precision: When the model returns similar words, they are truly related (few wrong words). Good for search engines to show relevant results.
• High recall: The model finds most related words, even if some are less relevant. Useful for creative writing tools that want many options.

Balancing these helps depending on the task: strict similarity or broad relatedness.

What "good" vs "bad" metric values look like

Good Word2Vec model:

• Cosine similarity between synonyms > 0.7
• Analogy accuracy > 70% on test questions
• Clear clusters of related words in visualization

Bad Word2Vec model:

• Cosine similarity near 0 for related words
• Analogy accuracy close to random guess (~25%)
• Random scatter in visualization with no clusters

Common pitfalls in Word2Vec metrics

• Overfitting: Model memorizes frequent word pairs but fails on new words.
• Data bias: If training data is small or biased, embeddings reflect that and give poor results.
• Ignoring context: Word2Vec gives one vector per word, so it can confuse words with multiple meanings.
• Misinterpreting cosine similarity: High similarity does not always mean synonyms; it can reflect related but different meanings.

Self-check question

Your Word2Vec model shows cosine similarity 0.95 between "cat" and "dog", but only 0.2 between "cat" and "kitten". Is this good? Why or why not?

Answer: This is not good. "Cat" and "kitten" are closely related (adult and baby), so similarity should be high. A low value means the model did not learn this relationship well. The high similarity between "cat" and "dog" might reflect they are both animals but is less precise than expected.

Key Result

Cosine similarity and analogy accuracy are key to check if Word2Vec captures meaningful word relationships.