Pre-trained embedding usage in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of embedding vector shape

What is the shape of the output embedding vector when using a pre-trained embedding layer with input size 10 and embedding dimension 50 for a batch of 3 samples?

NLP

import torch
import torch.nn as nn

embedding = nn.Embedding(num_embeddings=10, embedding_dim=50)
input_indices = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
output = embedding(input_indices)
print(output.shape)

A torch.Size([3, 3])

B torch.Size([10, 50])

C torch.Size([3, 50])

D torch.Size([3, 3, 50])

💡 Hint

Think about the input shape and how embedding layers map indices to vectors.

The input tensor has shape (3, 3) representing 3 samples each with 3 indices. The embedding layer converts each index to a 50-dimensional vector, so the output shape is (3, 3, 50).

❓ Model Choice

intermediate

Choosing pre-trained embeddings for sentiment analysis

You want to build a sentiment analysis model on movie reviews. Which pre-trained embedding is best suited to capture semantic meaning of words in this context?

A Pre-trained GloVe embeddings trained on Common Crawl

B Pre-trained embeddings from a speech recognition model

C One-hot encoded vectors

D Randomly initialized embeddings trained from scratch

💡 Hint

Consider embeddings trained on large text corpora relevant to general language.

GloVe embeddings trained on Common Crawl capture semantic relationships between words from a large text corpus, making them suitable for sentiment analysis tasks.

❓ Hyperparameter

advanced

Effect of freezing pre-trained embeddings

What is the effect of freezing the weights of a pre-trained embedding layer during training?

A The embedding weights are randomly re-initialized at each epoch

B The embedding weights are updated during training to adapt to the new task

C The embedding weights remain fixed and are not updated during training

D The embedding weights are discarded and replaced with one-hot vectors

💡 Hint

Freezing means preventing changes to the weights.

Freezing the embedding layer means its weights stay fixed and do not update during training, preserving the original pre-trained knowledge.

❓ Metrics

advanced

Evaluating embedding quality with downstream task accuracy

You compare two pre-trained embeddings by training the same classifier on a text classification task. Embedding A yields 85% accuracy, embedding B yields 78%. What can you conclude?

A Embedding B is better because lower accuracy means less overfitting

B Embedding A is better for this task because it leads to higher accuracy

C Both embeddings are equally good because accuracy differences are insignificant

D Embedding B is better because it has fewer parameters

💡 Hint

Higher accuracy usually means better feature representation for the task.

Higher accuracy indicates that embedding A provides better features for the classifier on this task compared to embedding B.

🔧 Debug

expert

Identifying error when loading pre-trained embeddings

You try to load pre-trained embeddings into your model but get a size mismatch error. What is the most likely cause?

NLP

import torch
import torch.nn as nn

embedding = nn.Embedding(num_embeddings=1000, embedding_dim=300)
pretrained_weights = torch.randn(500, 300)
embedding.weight.data.copy_(pretrained_weights)

A The pretrained weights have fewer embeddings (500) than the model expects (1000), causing size mismatch

B The pretrained weights tensor is not a float tensor

C The embedding dimension 300 does not match pretrained weights dimension 300

D The model's embedding layer is not initialized before copying weights

💡 Hint

Check the shape of pretrained weights vs model embedding weights.

The pretrained weights have 500 embeddings but the model expects 1000, so copying causes a size mismatch error.