Word2Vec (CBOW and Skip-gram) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Word2Vec (CBOW and Skip-gram)

Which metric matters for Word2Vec and WHY

For Word2Vec models like CBOW and Skip-gram, the main goal is to learn good word representations. We measure this by loss, which shows how well the model predicts context words. Lower loss means better word vectors.

Since Word2Vec is unsupervised, traditional accuracy or precision don't apply directly. Instead, we use intrinsic evaluation like cosine similarity between word vectors or analogy tests (e.g., "king" - "man" + "woman" ≈ "queen") to check quality.

In short, loss during training and semantic similarity in evaluation are key metrics.

Confusion matrix or equivalent visualization

Word2Vec does not use a confusion matrix because it predicts words in a large vocabulary, not classes. Instead, we visualize word vector quality with:

    Example analogy test:
    "king" - "man" + "woman" ≈ "queen"

    Cosine similarity matrix snippet:
    king queen man woman cat dog
    king    1.0   0.78  0.65  0.60  0.10 0.12
    queen   0.78  1.0   0.55  0.70  0.08 0.09
    man     0.65  0.55  1.0   0.50  0.05 0.07
    woman   0.60  0.70  0.50  1.0   0.06 0.08
    cat     0.10  0.08  0.05  0.06  1.0  0.85
    dog     0.12  0.09  0.07  0.08  0.85 1.0
    

This shows related words have higher similarity scores.

Precision vs Recall tradeoff (or equivalent) with examples

Word2Vec does not have precision or recall because it is not a classification model. Instead, there is a tradeoff between training speed and embedding quality:

• CBOW is faster and better for frequent words but may miss rare word nuances.
• Skip-gram is slower but better captures rare words and subtle meanings.

Choosing between CBOW and Skip-gram depends on your data and needs. For example, if you want fast training on common words, CBOW is good. For detailed rare word meaning, Skip-gram is better.

What "good" vs "bad" metric values look like for Word2Vec

Good Word2Vec model:

• Low training loss (e.g., steadily decreasing to a small value)
• High cosine similarity for related words (above 0.6 to 0.8)
• Correct answers on analogy tests (e.g., "king" - "man" + "woman" ≈ "queen")

Bad Word2Vec model:

• High or stagnant training loss (model not learning)
• Low cosine similarity even for related words (below 0.3)
• Poor analogy test results (random or wrong words)

Common pitfalls in Word2Vec metrics

• Ignoring rare words: CBOW may not learn good vectors for rare words, so evaluation should consider word frequency.
• Overfitting: Training loss very low but embeddings do not generalize well to new data.
• Data leakage: Using test data in training can inflate similarity scores.
• Misinterpreting loss: Loss alone does not guarantee semantic quality; always check analogy or similarity tests.

Self-check question

Your Word2Vec model has a low training loss but fails analogy tests and shows low cosine similarity for related words. Is it good? Why or why not?

Answer: No, it is not good. Low loss means the model fits training data, but poor analogy and similarity show it did not learn meaningful word relationships. You should check data quality, model parameters, or training process.

Key Result

For Word2Vec, low training loss plus high semantic similarity and good analogy test results indicate a good model.