Bird
Raised Fist0

Implement Word2Vec - Skip-Gram, Negative Sampling

CODINGadvancedInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/5
Input Word

The input word 'cat' is selected from the corpus.

🎯 Understanding the role of the input word in the Skip-Gram model.
input_word = 'cat'
context_words = ['sat', 'on', 'the', 'mat']
Context Words

Context words ['sat', 'on', 'the', 'mat'] are identified for 'cat'.

🎯 How context words are determined in the Skip-Gram model.
for context_word in context_words:
    # Process each context word
Negative Sampling

Negative samples ['dog', 'car', 'tree'] are randomly selected.

🎯 The importance of negative sampling in training efficiency.
negative_samples = ['dog', 'car', 'tree']
for neg_word in negative_samples:
    # Process each negative sample
Embedding Matrix

Vectors for 'cat' and context words are retrieved from the embedding matrix.

🎯 How embeddings are used to represent words numerically.
cat_vector = embedding_matrix['cat']
context_vectors = [embedding_matrix[word] for word in context_words]
Output Layer

Sigmoid activation is applied to predict context presence.

🎯 The role of the output layer in determining word relationships.
output = sigmoid(np.dot(cat_vector, context_vectors.T))
loss = -np.log(output)
📊
Diagram
Diagram
┌──────────────────────────────────────────────────────────────┐
│ Word2Vec Skip-Gram Model with Negative Sampling              │
├──────────────────────────────────────────────────────────────┤
│ Input Word: 'cat'                                            │
│ Context Words: ['sat', 'on', 'the', 'mat']                   │
│ Negative Samples: ['dog', 'car', 'tree']                     │
├──────────────────────────────────────────────────────────────┤
│ Embedding Matrix (Vocab x Embedding Size)                    │
│ ┌───────────────┐                                            │
│ │ 'cat' vector  │                                            │
│ │ 'sat' vector  │                                            │
│ │ ...           │                                            │
│ └───────────────┘                                            │
│                                                             │
│ Output Layer: Sigmoid Activation                            │
│ ┌───────────────┐                                            │
│ │ Positive Pair │                                            │
│ │ Negative Pair │                                            │
│ └───────────────┘                                            │
└──────────────────────────────────────────────────────────────┘
Labels
  • Input Word: The target word for which context is predicted
  • Context Words: Words surrounding the input word in the text
  • Negative Samples: Random words not in the context, used for training
  • Embedding Matrix: Contains vector representations of words
  • Output Layer: Uses sigmoid activation to predict context presence
🔑
Key Moments
▲
Q: Why is negative sampling used in the Skip-Gram model?
A: Negative sampling reduces computational complexity by only updating a small subset of weights.
Q: How does the embedding matrix facilitate word representation?
A: The embedding matrix maps words to dense vectors, capturing semantic relationships.
Q: What is the significance of the sigmoid activation in the output layer?
A: Sigmoid activation outputs probabilities, indicating the likelihood of context words.
Trace Summary

This diagram and execution trace illustrate the Word2Vec Skip-Gram model with Negative Sampling. The process begins with selecting an input word and identifying its context words. Negative samples are then chosen to improve training efficiency. The embedding matrix provides vector representations for words, which are used in the output layer with sigmoid activation to predict the presence of context words. This approach efficiently captures semantic relationships between words while reducing computational demands through negative sampling.

Primary Focus

✓ Implementing the Skip-Gram model with Negative Sampling in Word2Vec

Practice

(1/5)
1. What is the time complexity of applying Min-Max scaling to a dataset with n samples and m features?
easy
A. O(n * m)
B. O(m)
C. O(n^2)
D. O(n)

Solution

  1. Step 1: Understand Min-Max Scaling

    Min-Max scaling requires computing the min and max for each feature, which involves iterating over all samples for each feature.

  2. Step 2: Calculate Complexity

    For each of the m features, we iterate over n samples, resulting in O(n * m) complexity.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Min-Max scaling is O(n * m). [OK]
Hint: Min-Max scaling is O(n * m). [OK]
Common Mistakes:
2. What does the following Python code snippet output when implementing sinusoidal positional encoding?
import numpy as np

def positional_encoding(position, d_model):
    angle_rads = np.arange(position)[:, np.newaxis] / np.power(10000, (2 * (np.arange(d_model)[np.newaxis, :] // 2)) / np.float32(d_model))
    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
    return angle_rads

pe = positional_encoding(4, 4)
print(pe)
easy
A. A 4x4 matrix with alternating sine and cosine values
B. A 4x4 matrix with random values
C. A 4x4 identity matrix
D. A 4x4 matrix with all zeros

Solution

  1. Step 1: Understand the code logic

    The code generates a matrix where each row corresponds to a position and each column to a dimension, filled with sinusoidal values.

  2. Step 2: Eliminate wrong options

    A 4x4 matrix with all zeros is incorrect because the matrix is not filled with zeros. A 4x4 identity matrix is incorrect because the matrix is not an identity matrix. A 4x4 matrix with random values is incorrect because the values are not random but deterministic sinusoidal values.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sinusoidal encoding alternates sine/cosine. [OK]
Hint: Sinusoidal encoding alternates sine/cosine. [OK]
Common Mistakes:
3. 
import numpy as np

class DropoutLayer:
    def __init__(self, dropout_rate):
        self.dropout_rate = dropout_rate

    def forward(self, inputs, training=True):
        if training:
            self.mask = np.random.binomial(1, 1 - self.dropout_rate, size=inputs.shape)
            return inputs * self.mask
        else:
            return inputs * (1 - self.dropout_rate)

inputs = np.array([1.0, 2.0, 3.0])
dropout_layer = DropoutLayer(0.5)
output = dropout_layer.forward(inputs, training=True)
print(output)
Why might this implementation produce unexpected results during training?
medium
A. The dropout rate is too high.
B. The mask is not applied correctly.
C. The mask is not saved for backpropagation.
D. The inputs are not scaled during training.

Solution

  1. Step 1: Identify Training Mode Requirements

    During training, the mask must be saved for backpropagation to correctly compute gradients.

  2. Step 2: Eliminate Wrong Options

    The dropout rate is too high. is about hyperparameter tuning, not implementation. The mask is not applied correctly. is incorrect as the mask is applied correctly. The inputs are not scaled during training. is irrelevant during training.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Mask must be saved for backpropagation. [OK]
Hint: Mask must be saved for backpropagation. [OK]
Common Mistakes:
4. You implemented Robust Scaling but your model's performance dropped. What is the most likely cause?
medium
A. Robust Scaling is not suitable for normally distributed data.
B. Robust Scaling requires data to be centered around zero.
C. Robust Scaling scales data to a range of [0, 1], which is incorrect.
D. Robust Scaling removes outliers, which are important for your model.

Solution

  1. Step 1: Understand Robust Scaling

    Robust Scaling uses the median and IQR, which reduces the influence of outliers.

  2. Step 2: Identify the Issue

    If outliers are important, reducing their influence can harm model performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Robust Scaling reduces outlier impact. [OK]
Hint: Robust Scaling reduces outlier impact. [OK]
Common Mistakes:
5. Your mAP implementation is returning unusually high values. What is the most likely cause?
medium
A. Using precision instead of recall in calculations
B. Incorrect IoU threshold for positive matches
C. Not normalizing the precision values
D. Including false negatives in the precision calculation

Solution

  1. Step 1: Identify the Symptom

    High mAP values suggest too many matches are considered correct.

  2. Step 2: Analyze Each Option

    Incorrect IoU threshold for positive matches: A low IoU threshold can cause false positives to be counted as true positives, inflating mAP. Using precision instead of recall in calculations: Would affect recall, not mAP directly. Not normalizing the precision values: Would affect the scale, not the value. Including false negatives in the precision calculation: Would lower precision, not increase it.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Check IoU threshold for false positives [OK]
Hint: Check IoU threshold for false positives [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard