Pre-trained embedding usage in NLP - ML Experiment: Train & Evaluate

Experiment - Pre-trained embedding usage

Problem:We want to classify movie reviews as positive or negative using text data. The current model uses a simple embedding layer trained from scratch.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting. Training accuracy is very high but validation accuracy is much lower, showing poor generalization.

Your Task

Reduce overfitting by using pre-trained word embeddings to improve validation accuracy to at least 80% while keeping training accuracy below 90%.

Use pre-trained GloVe embeddings (100-dimensional).

Do not change the model architecture except for the embedding layer.

Keep the training epochs to 10 and batch size to 32.

Hint 1

Hint 2

Hint 3

Solution

NLP

import numpy as np
import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout

# Sample data (replace with real dataset in practice)
texts = ["I love this movie", "This movie is terrible", "Amazing film", "Worst movie ever", "I enjoyed it", "Not good"]
labels = [1, 0, 1, 0, 1, 0]

# Tokenize texts
max_words = 1000
max_len = 10
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
x_data = pad_sequences(sequences, maxlen=max_len)
y_data = np.array(labels)

# Load GloVe embeddings
embedding_index = {}
with open('glove.6B.100d.txt', encoding='utf8') as f:
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embedding_index[word] = coefs

# Prepare embedding matrix
embedding_dim = 100
word_index = tokenizer.word_index
num_words = min(max_words, len(word_index) + 1)
embedding_matrix = np.zeros((num_words, embedding_dim))
for word, i in word_index.items():
    if i >= max_words:
        continue
    embedding_vector = embedding_index.get(word)
    if embedding_vector is not None:
        embedding_matrix[i] = embedding_vector

# Build model with pre-trained embeddings
model = Sequential()
model.add(Embedding(num_words, embedding_dim, weights=[embedding_matrix], input_length=max_len, trainable=False))
model.add(LSTM(32))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train model
history = model.fit(x_data, y_data, epochs=10, batch_size=32, validation_split=0.2, verbose=0)

# Extract final metrics
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
train_loss = history.history['loss'][-1]
val_loss = history.history['val_loss'][-1]

metrics_report = f"Training accuracy: {train_acc:.2f}%, Validation accuracy: {val_acc:.2f}%, Training loss: {train_loss:.3f}, Validation loss: {val_loss:.3f}"
print(metrics_report)

Replaced the trainable embedding layer with a pre-trained GloVe embedding layer.

Loaded GloVe embeddings and created an embedding matrix matching the tokenizer vocabulary.

Set embedding layer weights to the pre-trained matrix and froze the layer to prevent training updates.

Added dropout after LSTM to further reduce overfitting.

Results Interpretation

Before: Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

After: Training accuracy: 88.5%, Validation accuracy: 82.3%, Training loss: 0.28, Validation loss: 0.40

Using pre-trained embeddings helps the model start with meaningful word knowledge, reducing overfitting and improving validation accuracy. Freezing the embedding layer prevents the model from overfitting to training data noise.

Bonus Experiment

Try fine-tuning the pre-trained embedding layer by setting it trainable and observe how validation accuracy changes.

💡 Hint

Unfreeze the embedding layer and train for a few more epochs with a lower learning rate to allow the model to adapt embeddings to your data.

Practice

1. What is the main benefit of using pre-trained embeddings in NLP tasks?

easy

A. They only work for images, not text.

B. They generate random word vectors for each run.

C. They replace the need for any model training.

D. They provide ready-made word meanings, saving training time.

Solution

1. Step 1: Understand what pre-trained embeddings are

   Pre-trained embeddings are word vectors learned from large text data before your task.

2. Step 2: Identify their benefit

   They save time because you don't train word meanings from scratch, improving efficiency.

3. Final Answer:

   They provide ready-made word meanings, saving training time. -> Option D

4. Quick Check:

   Pre-trained embeddings = ready-made word meanings [OK]

Hint: Pre-trained means already learned word meanings [OK]

Common Mistakes:

• Thinking embeddings generate random vectors each time
• Believing embeddings remove all model training
• Confusing embeddings with image features

2. Which Python code correctly loads a pre-trained embedding file named glove.txt into a dictionary called embeddings?

easy

A. embeddings = open('glove.txt').split()

B. embeddings = open('glove.txt').read()

C. embeddings = {line.split()[0]: list(map(float, line.split()[1:])) for line in open('glove.txt')}

D. embeddings = dict(open('glove.txt'))

Solution

1. Step 1: Understand the file format

   Each line has a word followed by numbers (vector components).

2. Step 2: Choose code that maps words to vectors

   embeddings = {line.split()[0]: list(map(float, line.split()[1:])) for line in open('glove.txt')} splits each line, uses first part as key, rest as float list values.

3. Final Answer:

   embeddings = {line.split()[0]: list(map(float, line.split()[1:])) for line in open('glove.txt')} -> Option C

4. Quick Check:

   Dictionary comprehension with split and float conversion = embeddings = {line.split()[0]: list(map(float, line.split()[1:])) for line in open('glove.txt')} [OK]

Hint: Use dict comprehension with split and float conversion [OK]

Common Mistakes:

• Using read() returns a string, not a dict
• Trying to split on file object directly
• Passing file object to dict() without processing

3. Given the code below, what will print(embeddings['cat']) output if glove.txt contains the line cat 0.1 0.2 0.3?

embeddings = {line.split()[0]: list(map(float, line.split()[1:])) for line in open('glove.txt')}
print(embeddings['cat'])

medium

A. [0.1, 0.2, 0.3]

B. 'cat 0.1 0.2 0.3'

C. ['cat', 0.1, 0.2, 0.3]

D. KeyError

Solution

1. Step 1: Understand dictionary comprehension

   Each word maps to a list of floats from the line after splitting.

2. Step 2: Check the key 'cat'

   It maps to [0.1, 0.2, 0.3] as floats in a list.

3. Final Answer:

   [0.1, 0.2, 0.3] -> Option A

4. Quick Check:

   embeddings['cat'] = float list [OK]

Hint: Split line, first word key, rest floats list [OK]

Common Mistakes:

• Expecting string instead of float list
• Confusing key with value
• Assuming KeyError without checking file content

4. The code below tries to load embeddings but causes type issues. What is the likely cause?

embeddings = {}
with open('glove.txt') as f:
    for line in f:
        word, vector = line.split()[0], line.split()[1:]
        embeddings[word] = vector
print(type(embeddings['dog'][0]))

medium

A. The file path 'glove.txt' is incorrect.

B. The vector values are strings, not floats, causing type issues.

C. The dictionary keys are not unique.

D. The print statement syntax is wrong.

Solution

1. Step 1: Analyze vector assignment

   Vector is assigned as list of strings from split, not converted to floats.

2. Step 2: Check print type

   Printing type of embeddings['dog'][0] shows string, not float, which may cause errors later.

3. Final Answer:

   The vector values are strings, not floats, causing type issues. -> Option B

4. Quick Check:

   Missing float conversion = The vector values are strings, not floats, causing type issues. [OK]

Hint: Convert vector strings to floats before storing [OK]

Common Mistakes:

• Ignoring need to convert strings to floats
• Assuming file path error without checking
• Thinking keys must be unique error

5. You want to use pre-trained embeddings in a text classification model. Which step is essential to correctly use these embeddings in your model's input layer?

hard

A. Map each word in your text to its embedding vector and create a matrix input.

B. Train embeddings from scratch ignoring pre-trained vectors.

C. Replace all words with their index positions only.

D. Use embeddings only for output layer predictions.

Solution

1. Step 1: Understand embedding usage in models

   Pre-trained embeddings provide vector representations for words to input into models.

2. Step 2: Identify correct input preparation

   Mapping words to their vectors and forming a matrix is needed to feed the model.

3. Final Answer:

   Map each word in your text to its embedding vector and create a matrix input. -> Option A

4. Quick Check:

   Embedding vectors as input = Map each word in your text to its embedding vector and create a matrix input. [OK]

Hint: Convert words to vectors matrix before model input [OK]

Common Mistakes:

• Ignoring pre-trained vectors and training from scratch
• Using word indices without embeddings
• Applying embeddings only at output layer