Word2Vec (CBOW and Skip-gram) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Word2Vec (CBOW and Skip-gram)

Which metric matters for Word2Vec and WHY

For Word2Vec models like CBOW and Skip-gram, the main goal is to learn good word representations. We measure this by loss, which shows how well the model predicts context words. Lower loss means better word vectors.

Since Word2Vec is unsupervised, traditional accuracy or precision don't apply directly. Instead, we use intrinsic evaluation like cosine similarity between word vectors or analogy tests (e.g., "king" - "man" + "woman" ≈ "queen") to check quality.

In short, loss during training and semantic similarity in evaluation are key metrics.

Confusion matrix or equivalent visualization

Word2Vec does not use a confusion matrix because it predicts words in a large vocabulary, not classes. Instead, we visualize word vector quality with:

    Example analogy test:
    "king" - "man" + "woman" ≈ "queen"

    Cosine similarity matrix snippet:
    king queen man woman cat dog
    king    1.0   0.78  0.65  0.60  0.10 0.12
    queen   0.78  1.0   0.55  0.70  0.08 0.09
    man     0.65  0.55  1.0   0.50  0.05 0.07
    woman   0.60  0.70  0.50  1.0   0.06 0.08
    cat     0.10  0.08  0.05  0.06  1.0  0.85
    dog     0.12  0.09  0.07  0.08  0.85 1.0
    

This shows related words have higher similarity scores.

Precision vs Recall tradeoff (or equivalent) with examples

Word2Vec does not have precision or recall because it is not a classification model. Instead, there is a tradeoff between training speed and embedding quality:

• CBOW is faster and better for frequent words but may miss rare word nuances.
• Skip-gram is slower but better captures rare words and subtle meanings.

Choosing between CBOW and Skip-gram depends on your data and needs. For example, if you want fast training on common words, CBOW is good. For detailed rare word meaning, Skip-gram is better.

What "good" vs "bad" metric values look like for Word2Vec

Good Word2Vec model:

• Low training loss (e.g., steadily decreasing to a small value)
• High cosine similarity for related words (above 0.6 to 0.8)
• Correct answers on analogy tests (e.g., "king" - "man" + "woman" ≈ "queen")

Bad Word2Vec model:

• High or stagnant training loss (model not learning)
• Low cosine similarity even for related words (below 0.3)
• Poor analogy test results (random or wrong words)

Common pitfalls in Word2Vec metrics

• Ignoring rare words: CBOW may not learn good vectors for rare words, so evaluation should consider word frequency.
• Overfitting: Training loss very low but embeddings do not generalize well to new data.
• Data leakage: Using test data in training can inflate similarity scores.
• Misinterpreting loss: Loss alone does not guarantee semantic quality; always check analogy or similarity tests.

Self-check question

Your Word2Vec model has a low training loss but fails analogy tests and shows low cosine similarity for related words. Is it good? Why or why not?

Answer: No, it is not good. Low loss means the model fits training data, but poor analogy and similarity show it did not learn meaningful word relationships. You should check data quality, model parameters, or training process.

Key Result

For Word2Vec, low training loss plus high semantic similarity and good analogy test results indicate a good model.

Practice

1. What is the main difference between the CBOW and Skip-gram models in Word2Vec?

easy

A. CBOW uses one-hot encoding, Skip-gram uses frequency encoding.

B. CBOW predicts a word based on its context, while Skip-gram predicts context words from a target word.

C. CBOW is used only for sentences, Skip-gram only for paragraphs.

D. CBOW requires labeled data, Skip-gram does not.

Solution

1. Step 1: Understand CBOW model purpose

   CBOW tries to predict the target word using the surrounding context words.

2. Step 2: Understand Skip-gram model purpose

   Skip-gram tries to predict the surrounding context words given the target word.

3. Final Answer:

   CBOW predicts a word based on its context, while Skip-gram predicts context words from a target word. -> Option B

4. Quick Check:

   CBOW = context to word, Skip-gram = word to context [OK]

Hint: Remember CBOW = context to word, Skip-gram = word to context [OK]

Common Mistakes:

• Confusing which model predicts context vs. target word
• Thinking both models do the same prediction
• Assuming CBOW needs labeled data

2. Which of the following is the correct way to initialize a Skip-gram Word2Vec model using the Gensim library in Python?

easy

A. Word2Vec(sentences, size=100, window=5, sg=0)

B. Word2Vec(sentences, vector_size=100, window=5, sg=0)

C. Word2Vec(sentences, size=100, window=5, sg=1)

D. Word2Vec(sentences, vector_size=100, window=5, sg=1)

Solution

1. Step 1: Identify correct parameter for Skip-gram

   In Gensim, 'sg=1' sets Skip-gram, 'sg=0' sets CBOW.

2. Step 2: Use correct parameter names

   Since Gensim 4.0+, 'vector_size' replaces 'size' for embedding dimension.

3. Final Answer:

   Word2Vec(sentences, vector_size=100, window=5, sg=1) -> Option D

4. Quick Check:

   sg=1 and vector_size used correctly [OK]

Hint: Use sg=1 for Skip-gram and vector_size for embedding size [OK]

Common Mistakes:

• Using 'size' instead of 'vector_size' in recent Gensim versions
• Setting sg=0 which is CBOW, not Skip-gram
• Confusing sg parameter values

3. Given the following code snippet using Gensim's Word2Vec with Skip-gram, what will be the output of model.wv.most_similar('king', topn=1) if the model is trained on a typical English corpus?

medium

A. [('run', similarity_score)]

B. [('apple', similarity_score)]

C. [('queen', similarity_score)]

D. [('car', similarity_score)]

Solution

1. Step 1: Understand Word2Vec similarity

   Word2Vec finds words with similar meanings or contexts; 'queen' is semantically close to 'king'.

2. Step 2: Analyze typical English corpus relations

   Words like 'apple', 'car', or 'run' are unrelated to 'king' in meaning or context.

3. Final Answer:

   [('queen', similarity_score)] -> Option C

4. Quick Check:

   Most similar to 'king' is 'queen' [OK]

Hint: Most similar to 'king' is usually 'queen' in English corpora [OK]

Common Mistakes:

• Choosing unrelated words as most similar
• Confusing syntactic similarity with semantic similarity
• Expecting exact similarity scores

4. You trained a CBOW Word2Vec model but get an error: KeyError: 'unknown_word' when querying model.wv['unknown_word']. What is the most likely cause and fix?

medium

A. The word was not in training data; retrain with larger corpus or check vocabulary before querying.

B. The model was trained with Skip-gram; switch to CBOW to fix.

C. The vector size is too small; increase vector_size parameter.

D. The window size is too large; reduce window parameter.

Solution

1. Step 1: Understand KeyError cause

   KeyError occurs when the queried word is not in the model's vocabulary.

2. Step 2: Fix by ensuring word presence

   Either add the word to training data or check if word exists before querying to avoid error.

3. Final Answer:

   The word was not in training data; retrain with larger corpus or check vocabulary before querying. -> Option A

4. Quick Check:

   KeyError means word missing in vocabulary [OK]

Hint: Check if word is in vocabulary before querying model vectors [OK]

Common Mistakes:

• Assuming model type (CBOW/Skip-gram) causes KeyError
• Changing vector or window size to fix missing word error
• Ignoring vocabulary check before querying

5. You want to train a Word2Vec model to capture rare word meanings better. Which approach is best?

hard

A. Use Skip-gram with a smaller window size and increase training epochs.

B. Use CBOW with a large window size and fewer epochs.

C. Use Skip-gram with a large window size and fewer epochs.

D. Use CBOW with a smaller window size and increase training epochs.

Solution

1. Step 1: Identify model for rare words

   Skip-gram is better at learning rare word representations than CBOW.

2. Step 2: Adjust window size and epochs

   Smaller window focuses on close context, improving rare word meaning; more epochs improve training quality.

3. Final Answer:

   Use Skip-gram with a smaller window size and increase training epochs. -> Option A

4. Quick Check:

   Skip-gram + small window + more epochs = better rare word capture [OK]

Hint: Skip-gram + small window + more epochs helps rare words [OK]

Common Mistakes:

• Choosing CBOW for rare word learning
• Using large window size which dilutes context
• Reducing epochs which limits training