Why embeddings capture semantic meaning in NLP - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why do word embeddings place similar words close together?

Word embeddings map words to vectors in space. Why do similar words end up close to each other in this space?

A Because embeddings assign random vectors initially and never update them.

B Because embeddings are trained to predict context words, so words used in similar contexts get similar vectors.

C Because embeddings group words by their length rather than meaning.

D Because embeddings only consider the first letter of each word.

💡 Hint

Think about how words that appear in similar sentences might share meaning.

Embeddings are learned by predicting words from their context or vice versa. This causes words used in similar contexts to have similar vectors, capturing semantic similarity.

❓ Predict Output

intermediate

Output of cosine similarity between embeddings

Given two word embeddings represented as vectors, what is the output of the cosine similarity calculation?

NLP

import numpy as np

vec1 = np.array([1, 2, 3])
vec2 = np.array([2, 4, 6])

cos_sim = np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
print('{:.2f}'.format(cos_sim))

A 0.87

B 0.00

C 1.00

D 0.50

💡 Hint

Consider if one vector is a scaled version of the other.

Cosine similarity measures the angle between vectors. If one vector is a positive multiple of the other, cosine similarity is 1.

❓ Model Choice

advanced

Choosing embedding size for semantic capture

You want to train word embeddings that capture rich semantic meaning. Which embedding size is most likely to work best?

A Embedding size of 50 dimensions

B Embedding size of 5000 dimensions

C Embedding size of 5 dimensions

D Embedding size of 1 dimension

💡 Hint

Think about balancing detail and overfitting.

Embedding sizes around 50 dimensions are common to capture semantic nuances without overfitting or excessive computation.

❓ Metrics

advanced

Evaluating semantic quality of embeddings

Which metric is best suited to evaluate if embeddings capture semantic similarity between words?

A Cosine similarity correlation with human similarity scores

B Mean Squared Error between embedding vectors

C Accuracy of a classification model using embeddings as input

D Number of unique words in the vocabulary

💡 Hint

Think about comparing embedding similarity to human judgments.

Cosine similarity correlation with human similarity scores measures how well embeddings reflect human semantic understanding.

🔧 Debug

expert

Why does this embedding training code produce identical vectors?

Consider this simplified embedding training snippet. Why do all word vectors end up identical?

NLP

import numpy as np

vocab = ['cat', 'dog', 'fish']
embeddings = {word: np.zeros(3) for word in vocab}

for epoch in range(3):
    for word in vocab:
        embeddings[word] += 0.1

print(embeddings)

A Because numpy arrays cannot be updated in-place.

B Because the loop only updates the first word's embedding.

C Because the code resets embeddings to zero inside the loop each time.

D Because all embeddings start at zero and are incremented by the same scalar, they remain identical vectors.

💡 Hint

Look at how the embeddings are initialized and updated.

All embeddings start as zero vectors and each is incremented by the same amount each iteration, so they remain identical.