Word2Vec (CBOW and Skip-gram) in NLP - Model Pipeline Trace

Model Pipeline - Word2Vec (CBOW and Skip-gram)

This pipeline trains a Word2Vec model to learn word meanings by looking at words around them. It uses two methods: CBOW predicts a word from its neighbors, and Skip-gram predicts neighbors from a word.

Data Flow - 5 Stages

1Raw Text Input

1000 sentences x variable length→Collect sentences from text corpus→1000 sentences x variable length

"The cat sat on the mat"

↓

2Tokenization

1000 sentences x variable length→Split sentences into words (tokens)→1000 sentences x variable length tokens

["The", "cat", "sat", "on", "the", "mat"]

↓

3Context Window Creation

1000 sentences x variable length tokens→Create pairs of target and context words using window size 2→Approx. 6000 word pairs (target, context)

Target: "sat", Context: ["The", "cat", "on", "the"]

↓

4One-hot Encoding

6000 word pairs→Convert words to one-hot vectors of vocabulary size 5000→6000 pairs of vectors (5000-dim each)

Target vector: [0,0,1,0,...], Context vectors: [[0,1,0,...], ...]

↓

5Model Training (CBOW or Skip-gram)

6000 pairs of vectors→Train neural network to predict target from context (CBOW) or context from target (Skip-gram)→Trained word embeddings matrix (5000 words x 100 dims)

Embedding vector for "cat": [0.12, -0.05, ..., 0.33]

Training Trace - Epoch by Epoch

Loss
5.0 |****
4.0 |*** 
3.0 |**  
2.0 |*   
1.0 |*   
    +----
     1 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	4.5	0.15	Initial loss high, accuracy low as model starts learning word relations
2	3.2	0.35	Loss decreases, accuracy improves as embeddings start capturing context
3	2.1	0.55	Model learns better word associations, accuracy rises
4	1.5	0.70	Loss continues to drop, embeddings become more meaningful
5	1.1	0.80	Training converges, good accuracy achieved

Prediction Trace - 5 Layers

Layer 1: Input Context Words (CBOW)

Layer 2: Embedding Layer

Layer 3: Average Embeddings

Layer 4: Output Layer (Softmax)

Layer 5: Prediction

Model Quiz - 3 Questions

In the CBOW model, what is the input to the neural network?

A The target word itself

B Context words around the target word

C Random noise vectors

D The entire sentence

Key Insight

Word2Vec learns word meanings by predicting words from their neighbors or vice versa. This helps capture relationships like synonyms or related concepts in a simple vector form.

Practice

1. What is the main difference between the CBOW and Skip-gram models in Word2Vec?

easy

A. CBOW uses one-hot encoding, Skip-gram uses frequency encoding.

B. CBOW predicts a word based on its context, while Skip-gram predicts context words from a target word.

C. CBOW is used only for sentences, Skip-gram only for paragraphs.

D. CBOW requires labeled data, Skip-gram does not.

Solution

1. Step 1: Understand CBOW model purpose

   CBOW tries to predict the target word using the surrounding context words.

2. Step 2: Understand Skip-gram model purpose

   Skip-gram tries to predict the surrounding context words given the target word.

3. Final Answer:

   CBOW predicts a word based on its context, while Skip-gram predicts context words from a target word. -> Option B

4. Quick Check:

   CBOW = context to word, Skip-gram = word to context [OK]

Hint: Remember CBOW = context to word, Skip-gram = word to context [OK]

Common Mistakes:

• Confusing which model predicts context vs. target word
• Thinking both models do the same prediction
• Assuming CBOW needs labeled data

2. Which of the following is the correct way to initialize a Skip-gram Word2Vec model using the Gensim library in Python?

easy

A. Word2Vec(sentences, size=100, window=5, sg=0)

B. Word2Vec(sentences, vector_size=100, window=5, sg=0)

C. Word2Vec(sentences, size=100, window=5, sg=1)

D. Word2Vec(sentences, vector_size=100, window=5, sg=1)

Solution

1. Step 1: Identify correct parameter for Skip-gram

   In Gensim, 'sg=1' sets Skip-gram, 'sg=0' sets CBOW.

2. Step 2: Use correct parameter names

   Since Gensim 4.0+, 'vector_size' replaces 'size' for embedding dimension.

3. Final Answer:

   Word2Vec(sentences, vector_size=100, window=5, sg=1) -> Option D

4. Quick Check:

   sg=1 and vector_size used correctly [OK]

Hint: Use sg=1 for Skip-gram and vector_size for embedding size [OK]

Common Mistakes:

• Using 'size' instead of 'vector_size' in recent Gensim versions
• Setting sg=0 which is CBOW, not Skip-gram
• Confusing sg parameter values

3. Given the following code snippet using Gensim's Word2Vec with Skip-gram, what will be the output of model.wv.most_similar('king', topn=1) if the model is trained on a typical English corpus?

medium

A. [('run', similarity_score)]

B. [('apple', similarity_score)]

C. [('queen', similarity_score)]

D. [('car', similarity_score)]

Solution

1. Step 1: Understand Word2Vec similarity

   Word2Vec finds words with similar meanings or contexts; 'queen' is semantically close to 'king'.

2. Step 2: Analyze typical English corpus relations

   Words like 'apple', 'car', or 'run' are unrelated to 'king' in meaning or context.

3. Final Answer:

   [('queen', similarity_score)] -> Option C

4. Quick Check:

   Most similar to 'king' is 'queen' [OK]

Hint: Most similar to 'king' is usually 'queen' in English corpora [OK]

Common Mistakes:

• Choosing unrelated words as most similar
• Confusing syntactic similarity with semantic similarity
• Expecting exact similarity scores

4. You trained a CBOW Word2Vec model but get an error: KeyError: 'unknown_word' when querying model.wv['unknown_word']. What is the most likely cause and fix?

medium

A. The word was not in training data; retrain with larger corpus or check vocabulary before querying.

B. The model was trained with Skip-gram; switch to CBOW to fix.

C. The vector size is too small; increase vector_size parameter.

D. The window size is too large; reduce window parameter.

Solution

1. Step 1: Understand KeyError cause

   KeyError occurs when the queried word is not in the model's vocabulary.

2. Step 2: Fix by ensuring word presence

   Either add the word to training data or check if word exists before querying to avoid error.

3. Final Answer:

   The word was not in training data; retrain with larger corpus or check vocabulary before querying. -> Option A

4. Quick Check:

   KeyError means word missing in vocabulary [OK]

Hint: Check if word is in vocabulary before querying model vectors [OK]

Common Mistakes:

• Assuming model type (CBOW/Skip-gram) causes KeyError
• Changing vector or window size to fix missing word error
• Ignoring vocabulary check before querying

5. You want to train a Word2Vec model to capture rare word meanings better. Which approach is best?

hard

A. Use Skip-gram with a smaller window size and increase training epochs.

B. Use CBOW with a large window size and fewer epochs.

C. Use Skip-gram with a large window size and fewer epochs.

D. Use CBOW with a smaller window size and increase training epochs.

Solution

1. Step 1: Identify model for rare words

   Skip-gram is better at learning rare word representations than CBOW.

2. Step 2: Adjust window size and epochs

   Smaller window focuses on close context, improving rare word meaning; more epochs improve training quality.

3. Final Answer:

   Use Skip-gram with a smaller window size and increase training epochs. -> Option A

4. Quick Check:

   Skip-gram + small window + more epochs = better rare word capture [OK]

Hint: Skip-gram + small window + more epochs helps rare words [OK]

Common Mistakes:

• Choosing CBOW for rare word learning
• Using large window size which dilutes context
• Reducing epochs which limits training