Pre-trained embeddings (GloVe) in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What are pre-trained embeddings like GloVe in simple terms?

Pre-trained embeddings are like a dictionary where each word is turned into a list of numbers that capture its meaning. GloVe is one popular set of these number lists made by looking at lots of text.

beginner

Why use pre-trained embeddings instead of learning from scratch?

Using pre-trained embeddings saves time and helps the model understand words better because they have already learned word meanings from a huge amount of text.

intermediate

How does GloVe create word embeddings?

GloVe looks at how often words appear together in big text collections and uses math to turn these patterns into number lists that show word meanings.

intermediate

What is the shape of a GloVe embedding matrix used in TensorFlow?

The embedding matrix shape is (vocabulary size, embedding dimension). Each row is a word's vector, for example, (10000, 100) means 10,000 words each with 100 numbers.

intermediate

How do you load GloVe embeddings into a TensorFlow Embedding layer?

You first create a matrix with GloVe vectors for your vocabulary, then set this matrix as weights in a TensorFlow Embedding layer and freeze it so it doesn't change during training.

What does each number in a GloVe embedding vector represent?

A A feature capturing some aspect of the word's meaning

B The frequency of the word in the text

C The length of the word

D The position of the word in a sentence

Each number in the embedding vector captures a feature related to the word's meaning based on word co-occurrence.

Why might you freeze the embedding layer weights when using pre-trained GloVe embeddings?

A To reduce the size of the model

B To speed up training by skipping the embedding layer

C To allow the embeddings to learn new meanings

D To keep the learned word meanings unchanged during training

Freezing the embedding layer keeps the pre-trained word meanings fixed, preventing them from changing during training.

What is the main source of data GloVe uses to learn embeddings?

A Audio recordings of speech

B Images paired with words

C Word co-occurrence statistics from large text corpora

D Randomly generated numbers

GloVe uses word co-occurrence counts from large text collections to learn word embeddings.

If your vocabulary has 5000 words and you use 50-dimensional GloVe vectors, what is the shape of the embedding matrix?

A (50, 5000)

B (5000, 50)

C (5000, 5000)

D (50, 50)

The embedding matrix shape is (vocabulary size, embedding dimension), so (5000, 50).

Which of these is NOT a benefit of using pre-trained embeddings like GloVe?

A They guarantee perfect understanding of all words

B They provide a good starting point for word meanings

C They reduce the need for large labeled datasets

D They speed up model training

Pre-trained embeddings help but do not guarantee perfect understanding of all words.

Explain in your own words what pre-trained embeddings like GloVe are and why they are useful.

Think about how a dictionary helps you understand words.

Describe the steps to use GloVe embeddings in a TensorFlow model.

Consider how you prepare and plug in the word vectors.

Practice

1. What is the main purpose of using pre-trained GloVe embeddings in TensorFlow models?

easy

A. To convert words into meaningful number vectors using prior knowledge

B. To generate random numbers for model initialization

C. To increase the size of the training dataset

D. To replace the output layer of the model

Solution

1. Step 1: Understand what embeddings do

   Embeddings convert words into vectors that capture meaning from large text data.

2. Step 2: Identify the role of pre-trained GloVe embeddings

   GloVe embeddings provide these vectors based on prior training on big text corpora, helping models understand word meanings without training from scratch.

3. Final Answer:

   To convert words into meaningful number vectors using prior knowledge -> Option A

4. Quick Check:

   Pre-trained embeddings = meaningful word vectors [OK]

Hint: Pre-trained embeddings map words to vectors using learned knowledge [OK]

Common Mistakes:

• Thinking embeddings generate random numbers
• Confusing embeddings with dataset size
• Assuming embeddings replace output layers

2. Which TensorFlow code snippet correctly loads GloVe embeddings into an Embedding layer with fixed weights?

easy

A. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)

B. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer='random_uniform', trainable=True)

C. embedding_layer = tf.keras.layers.Dense(units=embedding_dim, activation='relu')

D. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.RandomNormal(), trainable=False)

Solution

1. Step 1: Identify correct Embedding layer usage for GloVe

   We use Embedding layer with embeddings_initializer set to Constant(glove_matrix) to load pre-trained weights.

2. Step 2: Check trainable flag for fixed embeddings

   Setting trainable=False ensures embeddings are not updated during training, which matches the question.

3. Final Answer:

   embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False) -> Option A

4. Quick Check:

   Constant initializer + trainable=False = fixed GloVe embeddings [OK]

Hint: Use Constant initializer and trainable=False for fixed GloVe embeddings [OK]

Common Mistakes:

• Using random initializers instead of Constant
• Setting trainable=True when embeddings should be fixed
• Using Dense layer instead of Embedding

3. Given the code below, what will be the shape of embedding_output?

embedding_layer = tf.keras.layers.Embedding(input_dim=10000, output_dim=50, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)
input_seq = tf.constant([[1, 5, 10], [2, 3, 4]])
embedding_output = embedding_layer(input_seq)

medium

A. (2, 50)

B. (3, 2, 50)

C. (2, 3, 50)

D. (3, 50)

Solution

1. Step 1: Analyze input sequence shape

   Input sequence has shape (2, 3) because it has 2 sequences each with 3 tokens.

2. Step 2: Understand Embedding output shape

   Embedding layer converts each token to a 50-dimensional vector, so output shape is (2, 3, 50).

3. Final Answer:

   (2, 3, 50) -> Option C

4. Quick Check:

   Input shape (2,3) + embedding_dim 50 = (2,3,50) [OK]

Hint: Embedding output shape = input shape + embedding dimension [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Forgetting embedding dimension in output shape
• Assuming output shape is 2D instead of 3D

4. Identify the error in this code snippet that loads GloVe embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix))
embedding_layer.trainable = False
model = tf.keras.Sequential([embedding_layer, tf.keras.layers.Dense(10)])

medium

A. Dense layer cannot follow an Embedding layer

B. Using Constant initializer is incorrect for GloVe embeddings

C. Embedding layer must be the last layer in the model

D. Setting trainable attribute after layer creation does not freeze embeddings

Solution

1. Step 1: Check how to freeze embeddings

   Setting trainable=False must be done during layer creation, not by assigning attribute later.

2. Step 2: Understand effect of late trainable assignment

   Assigning embedding_layer.trainable = False after creation does not properly freeze weights in Keras models.

3. Final Answer:

   Setting trainable attribute after layer creation does not freeze embeddings -> Option D

4. Quick Check:

   Freeze embeddings by trainable=False in constructor [OK]

Hint: Set trainable=False when creating Embedding layer to freeze weights [OK]

Common Mistakes:

• Setting trainable attribute after layer creation
• Thinking Constant initializer is wrong for GloVe
• Believing Embedding must be last layer

5. You want to fine-tune GloVe embeddings on a small dataset. Which approach is best to avoid overfitting while still improving embeddings?

hard

A. Randomly initialize embeddings and keep trainable=False

B. Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings

C. Use GloVe embeddings with trainable=False and no dropout

D. Freeze embeddings and add many Dense layers without regularization

Solution

1. Step 1: Understand fine-tuning embeddings on small data

   Fine-tuning means allowing embeddings to update (trainable=True) but small data risks overfitting.

2. Step 2: Use dropout to reduce overfitting

   Applying dropout after embeddings helps regularize and prevent overfitting during fine-tuning.

3. Final Answer:

   Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings -> Option B

4. Quick Check:

   Fine-tune + dropout = better small data training [OK]

Hint: Fine-tune with trainable=True plus dropout to avoid overfitting [OK]

Common Mistakes:

• Freezing embeddings when fine-tuning is needed
• Skipping dropout causing overfitting
• Random initialization losing pre-trained knowledge