Pre-trained embeddings (GloVe) in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Pre-trained embeddings (GloVe)

Which metric matters for Pre-trained embeddings (GloVe) and WHY

When using pre-trained embeddings like GloVe, the main goal is to improve how well your model understands words. Metrics like accuracy or loss on your specific task (e.g., text classification) show if the embeddings help your model make better predictions. Since embeddings help represent words, the quality of these embeddings affects the final task's performance. So, focus on task-specific metrics such as accuracy, precision, recall, or F1 score depending on your problem.

Confusion matrix example

Confusion Matrix for a text classification task using GloVe embeddings:

          Predicted
          Pos   Neg
Actual Pos  85    15
       Neg  10    90

- True Positives (TP) = 85
- False Positives (FP) = 10
- True Negatives (TN) = 90
- False Negatives (FN) = 15

Total samples = 85 + 10 + 90 + 15 = 200

Precision = TP / (TP + FP) = 85 / (85 + 10) = 0.8947
Recall = TP / (TP + FN) = 85 / (85 + 15) = 0.85
F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.871
    

Precision vs Recall tradeoff with examples

Using GloVe embeddings can affect how your model balances precision and recall.

• High Precision: Your model is very sure when it predicts a positive. For example, in spam detection, you want high precision so you don't mark good emails as spam.
• High Recall: Your model finds most of the positive cases. For example, in medical diagnosis, you want high recall to catch as many sick patients as possible.

Depending on your task, you might tune your model to favor precision or recall. GloVe embeddings help by providing rich word meanings, which can improve both metrics if used well.

What "good" vs "bad" metric values look like for GloVe embeddings use

Good metrics mean your model understands text well thanks to embeddings:

• Good: Accuracy > 80%, Precision and Recall both above 80%, F1 score near or above 0.8.
• Bad: Accuracy below 60%, Precision or Recall very low (below 50%), F1 score below 0.5.

If your metrics are bad, it might mean the embeddings are not helping or your model needs more tuning.

Common pitfalls when evaluating models with GloVe embeddings

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% of data is negative, a model predicting all negatives gets 90% accuracy but is useless.
• Data leakage: Using test data during training can inflate metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means your model memorizes training data but does not generalize.
• Ignoring task metrics: Using only accuracy when recall or precision matters can hide problems.

Self-check question

Your model using GloVe embeddings has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the recall is very low. This means the model misses most fraud cases, which is dangerous because catching fraud is critical. You want high recall to find as many frauds as possible, even if it means some false alarms.

Key Result

Pre-trained embeddings like GloVe improve task metrics such as accuracy, precision, and recall, which reflect how well the model understands text.

Practice

1. What is the main purpose of using pre-trained GloVe embeddings in TensorFlow models?

easy

A. To convert words into meaningful number vectors using prior knowledge

B. To generate random numbers for model initialization

C. To increase the size of the training dataset

D. To replace the output layer of the model

Solution

1. Step 1: Understand what embeddings do

   Embeddings convert words into vectors that capture meaning from large text data.

2. Step 2: Identify the role of pre-trained GloVe embeddings

   GloVe embeddings provide these vectors based on prior training on big text corpora, helping models understand word meanings without training from scratch.

3. Final Answer:

   To convert words into meaningful number vectors using prior knowledge -> Option A

4. Quick Check:

   Pre-trained embeddings = meaningful word vectors [OK]

Hint: Pre-trained embeddings map words to vectors using learned knowledge [OK]

Common Mistakes:

• Thinking embeddings generate random numbers
• Confusing embeddings with dataset size
• Assuming embeddings replace output layers

2. Which TensorFlow code snippet correctly loads GloVe embeddings into an Embedding layer with fixed weights?

easy

A. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)

B. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer='random_uniform', trainable=True)

C. embedding_layer = tf.keras.layers.Dense(units=embedding_dim, activation='relu')

D. embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.RandomNormal(), trainable=False)

Solution

1. Step 1: Identify correct Embedding layer usage for GloVe

   We use Embedding layer with embeddings_initializer set to Constant(glove_matrix) to load pre-trained weights.

2. Step 2: Check trainable flag for fixed embeddings

   Setting trainable=False ensures embeddings are not updated during training, which matches the question.

3. Final Answer:

   embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False) -> Option A

4. Quick Check:

   Constant initializer + trainable=False = fixed GloVe embeddings [OK]

Hint: Use Constant initializer and trainable=False for fixed GloVe embeddings [OK]

Common Mistakes:

• Using random initializers instead of Constant
• Setting trainable=True when embeddings should be fixed
• Using Dense layer instead of Embedding

3. Given the code below, what will be the shape of embedding_output?

embedding_layer = tf.keras.layers.Embedding(input_dim=10000, output_dim=50, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix), trainable=False)
input_seq = tf.constant([[1, 5, 10], [2, 3, 4]])
embedding_output = embedding_layer(input_seq)

medium

A. (2, 50)

B. (3, 2, 50)

C. (2, 3, 50)

D. (3, 50)

Solution

1. Step 1: Analyze input sequence shape

   Input sequence has shape (2, 3) because it has 2 sequences each with 3 tokens.

2. Step 2: Understand Embedding output shape

   Embedding layer converts each token to a 50-dimensional vector, so output shape is (2, 3, 50).

3. Final Answer:

   (2, 3, 50) -> Option C

4. Quick Check:

   Input shape (2,3) + embedding_dim 50 = (2,3,50) [OK]

Hint: Embedding output shape = input shape + embedding dimension [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Forgetting embedding dimension in output shape
• Assuming output shape is 2D instead of 3D

4. Identify the error in this code snippet that loads GloVe embeddings:

embedding_layer = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, embeddings_initializer=tf.keras.initializers.Constant(glove_matrix))
embedding_layer.trainable = False
model = tf.keras.Sequential([embedding_layer, tf.keras.layers.Dense(10)])

medium

A. Dense layer cannot follow an Embedding layer

B. Using Constant initializer is incorrect for GloVe embeddings

C. Embedding layer must be the last layer in the model

D. Setting trainable attribute after layer creation does not freeze embeddings

Solution

1. Step 1: Check how to freeze embeddings

   Setting trainable=False must be done during layer creation, not by assigning attribute later.

2. Step 2: Understand effect of late trainable assignment

   Assigning embedding_layer.trainable = False after creation does not properly freeze weights in Keras models.

3. Final Answer:

   Setting trainable attribute after layer creation does not freeze embeddings -> Option D

4. Quick Check:

   Freeze embeddings by trainable=False in constructor [OK]

Hint: Set trainable=False when creating Embedding layer to freeze weights [OK]

Common Mistakes:

• Setting trainable attribute after layer creation
• Thinking Constant initializer is wrong for GloVe
• Believing Embedding must be last layer

5. You want to fine-tune GloVe embeddings on a small dataset. Which approach is best to avoid overfitting while still improving embeddings?

hard

A. Randomly initialize embeddings and keep trainable=False

B. Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings

C. Use GloVe embeddings with trainable=False and no dropout

D. Freeze embeddings and add many Dense layers without regularization

Solution

1. Step 1: Understand fine-tuning embeddings on small data

   Fine-tuning means allowing embeddings to update (trainable=True) but small data risks overfitting.

2. Step 2: Use dropout to reduce overfitting

   Applying dropout after embeddings helps regularize and prevent overfitting during fine-tuning.

3. Final Answer:

   Initialize Embedding layer with GloVe weights, set trainable=True, and use dropout after embeddings -> Option B

4. Quick Check:

   Fine-tune + dropout = better small data training [OK]

Hint: Fine-tune with trainable=True plus dropout to avoid overfitting [OK]

Common Mistakes:

• Freezing embeddings when fine-tuning is needed
• Skipping dropout causing overfitting
• Random initialization losing pre-trained knowledge