Pre-trained embeddings (GloVe) in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the shape of the embedding matrix after loading GloVe vectors?

Given a vocabulary size of 10000 and embedding dimension of 50, after loading GloVe embeddings into a TensorFlow Embedding layer, what will be the shape of the embedding matrix?

TensorFlow

import numpy as np
import tensorflow as tf
vocab_size = 10000
embedding_dim = 50
embedding_matrix = np.zeros((vocab_size, embedding_dim))
embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, weights=[embedding_matrix], trainable=False)
print(embedding_layer.weights[0].shape)

A (10000, 10000)

B (50, 10000)

C (10000, 50)

D (50, 50)

💡 Hint

Remember the embedding matrix shape is (vocab_size, embedding_dim).

The embedding matrix has one row per word in the vocabulary and one column per embedding dimension. So shape is (10000, 50).

❓ Model Choice

intermediate

Which TensorFlow layer correctly uses pre-trained GloVe embeddings?

You want to use pre-trained GloVe embeddings in a TensorFlow model. Which layer setup correctly loads the embeddings and prevents them from updating during training?

A tf.keras.layers.Embedding(input_dim=10000, output_dim=50, weights=[embedding_matrix], trainable=False)

B tf.keras.layers.Embedding(input_dim=10000, output_dim=50, trainable=True)

C tf.keras.layers.Dense(50, activation='relu', weights=[embedding_matrix])

D tf.keras.layers.Embedding(input_dim=50, output_dim=10000, weights=[embedding_matrix], trainable=False)

💡 Hint

Embedding layer input_dim is vocab size, output_dim is embedding size.

Option A correctly sets input_dim and output_dim, loads weights, and disables training to keep embeddings fixed.

❓ Hyperparameter

advanced

What is the effect of setting trainable=True on a pre-trained GloVe embedding layer?

If you set trainable=True on a TensorFlow Embedding layer initialized with pre-trained GloVe embeddings, what happens during training?

A The embeddings will remain fixed and not change.

B The embeddings will be updated and fine-tuned to the specific task.

C The model will ignore the embeddings and use random vectors.

D The embeddings will be deleted before training starts.

💡 Hint

trainable=True means weights can change during backpropagation.

Setting trainable=True allows the model to adjust the embeddings to better fit the training data.

🔧 Debug

advanced

Why does this code raise a ValueError when loading GloVe embeddings?

Consider this code snippet: embedding_dim = 100 embedding_matrix = np.zeros((vocab_size, embedding_dim)) for word, i in word_index.items(): embedding_vector = embeddings_index.get(word) if embedding_vector is not None: embedding_matrix[i] = embedding_vector It raises a ValueError: could not broadcast input array from shape (50,) into shape (100,). Why?

A The loop tries to assign a string instead of a numeric vector.

B The word_index dictionary is empty, so no embeddings are assigned.

C embedding_matrix is initialized with wrong data type causing mismatch.

D The GloVe embeddings loaded have dimension 50, but embedding_matrix expects 100.

💡 Hint

Check the dimension of the loaded GloVe vectors vs embedding_matrix.

The error occurs because the GloVe vectors have 50 dimensions but embedding_matrix expects 100 dimensions per vector.

🧠 Conceptual

expert

Why use pre-trained GloVe embeddings instead of training embeddings from scratch?

Which of the following is the best reason to use pre-trained GloVe embeddings in a natural language processing model?

A They provide rich semantic word representations learned from large corpora, improving model performance especially with limited data.

B They guarantee 100% accuracy on any NLP task without further training.

C They reduce the model size by compressing all words into a single vector.

D They eliminate the need for tokenization and preprocessing of text data.

💡 Hint

Think about what pre-trained embeddings capture from language data.

Pre-trained embeddings capture semantic relationships from large text corpora, helping models generalize better especially when training data is small.

Practice

1. What is the main purpose of using pre-trained GloVe embeddings in TensorFlow models?

easy

A. To convert words into meaningful number vectors using prior knowledge

B. To generate random numbers for model initialization

C. To increase the size of the training dataset

D. To replace the output layer of the model

Solution

1. Step 1: Understand what embeddings do

   Embeddings convert words into vectors that capture meaning from large text data.

2. Step 2: Identify the role of pre-trained GloVe embeddings

   GloVe embeddings provide these vectors based on prior training on big text corpora, helping models understand word meanings without training from scratch.

3. Final Answer:

   To convert words into meaningful number vectors using prior knowledge -> Option A

4. Quick Check:

   Pre-trained embeddings = meaningful word vectors [OK]

Hint: Pre-trained embeddings map words to vectors using learned knowledge [OK]

Common Mistakes:

• Thinking embeddings generate random numbers
• Confusing embeddings with dataset size
• Assuming embeddings replace output layers

2. Which TensorFlow code snippet correctly loads GloVe embeddings into an Embedding layer with fixed weights?

easy

A. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)

B. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer='random_uniform', trainable=True)

C. embedding_layer = tf.keras.layers.Dense(units=embedding_dim, activation='relu')

D. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.RandomNormal(), trainable=False)

Solution

1. Step 1: Identify correct Embedding layer usage for GloVe

   We use Embedding layer with embeddings_initializer set to Constant(glove_matrix) to load pre-trained weights.

2. Step 2: Check trainable flag for fixed embeddings

   Setting trainable=False ensures embeddings are not updated during training, which matches the question.

3. Final Answer:

   embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False) -> Option A

4. Quick Check:

   Constant initializer + trainable=False = fixed GloVe embeddings [OK]

Hint: Use Constant initializer and trainable=False for fixed GloVe embeddings [OK]

Common Mistakes:

• Using random initializers instead of Constant
• Setting trainable=True when embeddings should be fixed
• Using Dense layer instead of Embedding

3. Given the code below, what will be the shape of embedding_output?

embedding_layer = tf.keras.layers.Embedding(input_dim=10000, output_dim=50, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)
input_seq = tf.constant([[1, 5, 10], [2, 3, 4]])
embedding_output = embedding_layer(input_seq)

medium

A. (2, 50)

B. (3, 2, 50)

C. (2, 3, 50)

D. (3, 50)

Solution

1. Step 1: Analyze input sequence shape

   Input sequence has shape (2, 3) because it has 2 sequences each with 3 tokens.

2. Step 2: Understand Embedding output shape

   Embedding layer converts each token to a 50-dimensional vector, so output shape is (2, 3, 50).

3. Final Answer:

   (2, 3, 50) -> Option C

4. Quick Check:

   Input shape (2,3) + embedding_dim 50 = (2,3,50) [OK]

Hint: Embedding output shape = input shape + embedding dimension [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Forgetting embedding dimension in output shape
• Assuming output shape is 2D instead of 3D

4. Identify the error in this code snippet that loads GloVe embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix))
embedding_layer.trainable = False
model = tf.keras.Sequential([embedding_layer, tf.keras.layers.Dense(10)])

medium

A. Dense layer cannot follow an Embedding layer

B. Using Constant initializer is incorrect for GloVe embeddings

C. Embedding layer must be the last layer in the model

D. Setting trainable attribute after layer creation does not freeze embeddings

Solution

1. Step 1: Check how to freeze embeddings

   Setting trainable=False must be done during layer creation, not by assigning attribute later.

2. Step 2: Understand effect of late trainable assignment

   Assigning embedding_layer.trainable = False after creation does not properly freeze weights in Keras models.

3. Final Answer:

   Setting trainable attribute after layer creation does not freeze embeddings -> Option D

4. Quick Check:

   Freeze embeddings by trainable=False in constructor [OK]

Hint: Set trainable=False when creating Embedding layer to freeze weights [OK]

Common Mistakes:

• Setting trainable attribute after layer creation
• Thinking Constant initializer is wrong for GloVe
• Believing Embedding must be last layer

5. You want to fine-tune GloVe embeddings on a small dataset. Which approach is best to avoid overfitting while still improving embeddings?

hard

A. Randomly initialize embeddings and keep trainable=False

B. Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings

C. Use GloVe embeddings with trainable=False and no dropout

D. Freeze embeddings and add many Dense layers without regularization

Solution

1. Step 1: Understand fine-tuning embeddings on small data

   Fine-tuning means allowing embeddings to update (trainable=True) but small data risks overfitting.

2. Step 2: Use dropout to reduce overfitting

   Applying dropout after embeddings helps regularize and prevent overfitting during fine-tuning.

3. Final Answer:

   Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings -> Option B

4. Quick Check:

   Fine-tune + dropout = better small data training [OK]

Hint: Fine-tune with trainable=True plus dropout to avoid overfitting [OK]

Common Mistakes:

• Freezing embeddings when fine-tuning is needed
• Skipping dropout causing overfitting
• Random initialization losing pre-trained knowledge