Embedding layer usage in NLP - ML Experiment: Train & Evaluate

Experiment - Embedding layer usage

Problem:We want to classify movie reviews as positive or negative using a neural network. Currently, the model uses one-hot encoding for words, which creates very large input vectors and trains slowly.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%, Training loss: 0.25, Validation loss: 0.65

Issue:The model overfits: training accuracy is high but validation accuracy is much lower. Also, one-hot encoding wastes memory and does not capture word meaning.

Your Task

Replace one-hot encoding with an embedding layer to reduce overfitting and improve validation accuracy to at least 80%. Keep training accuracy below 90% to avoid overfitting.

Use the same dataset and model architecture except for input encoding.

Do not increase the number of training epochs beyond 10.

Keep batch size at 32.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Flatten, Dense, Dropout

# Load data
max_features = 10000  # number of words to consider
maxlen = 100  # cut texts after this number of words

(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)

# Pad sequences to same length
X_train = pad_sequences(X_train, maxlen=maxlen)
X_test = pad_sequences(X_test, maxlen=maxlen)

# Build model with embedding layer
model = Sequential([
    Embedding(input_dim=max_features, output_dim=50, input_length=maxlen),
    Dropout(0.3),
    Flatten(),
    Dense(32, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=10, batch_size=32, validation_split=0.2, verbose=2)

# Evaluate on test data
results = model.evaluate(X_test, y_test, verbose=0)

print(f'Test loss: {results[0]:.3f}, Test accuracy: {results[1]*100:.2f}%')

Replaced one-hot encoded input with integer sequences padded to fixed length.

Added an Embedding layer as the first layer to learn word representations.

Added Dropout after embedding to reduce overfitting.

Kept the rest of the model architecture similar.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 75%, high overfitting.

After: Training accuracy 88%, Validation accuracy 82%, better generalization.

Using an embedding layer helps the model learn meaningful word features in a smaller space, reducing overfitting and improving validation accuracy.

Bonus Experiment

Try adding a recurrent layer (like LSTM) after the embedding layer to capture word order and see if validation accuracy improves further.

💡 Hint

Use tf.keras.layers.LSTM with 32 units after the embedding and dropout layers.

Practice

1. What is the main purpose of an Embedding layer in NLP models?

easy

A. To split sentences into individual characters

B. To count the number of words in a sentence

C. To convert words into dense vectors that capture meaning

D. To remove stop words from text

Solution

1. Step 1: Understand what embedding layers do

   Embedding layers transform words or tokens into dense numeric vectors that represent semantic meaning.

2. Step 2: Compare options with embedding purpose

   Counting words, removing stop words, or splitting characters are preprocessing steps, not embedding functions.

3. Final Answer:

   To convert words into dense vectors that capture meaning -> Option C

4. Quick Check:

   Embedding = word vectors [OK]

Hint: Embedding layers create numeric word meanings [OK]

Common Mistakes:

• Confusing embedding with tokenization
• Thinking embedding counts words
• Assuming embedding removes words

2. Which of the following is the correct way to create an embedding layer in TensorFlow Keras for 1000 words with 50 dimensions?

easy

A. Embedding(input_dim=1000, output_dim=50)

B. Embedding(output_dim=1000, input_dim=50)

C. Embedding(input_dim=50, output_dim=1000)

D. Embedding(1000, 100)

Solution

1. Step 1: Recall embedding layer parameters

   The first parameter input_dim is vocabulary size (1000), second output_dim is embedding size (50).

2. Step 2: Match parameters to options

   Only Embedding(input_dim=1000, output_dim=50) has the correct parameters: input_dim as vocabulary size (1000) and output_dim as embedding dimension (50). The others either swap these values or use incorrect dimensions.

3. Final Answer:

   Embedding(input_dim=1000, output_dim=50) -> Option A

4. Quick Check:

   input_dim = vocab size, output_dim = vector size [OK]

Hint: input_dim = vocab size, output_dim = vector size [OK]

Common Mistakes:

• Swapping input_dim and output_dim
• Using wrong parameter order
• Confusing embedding size with vocab size

3. Given the code below, what is the shape of the output tensor after the embedding layer?

import tensorflow as tf
embedding = tf.keras.layers.Embedding(input_dim=5000, output_dim=16)
input_seq = tf.constant([[1, 2, 3], [4, 5, 6]])
output = embedding(input_seq)
print(output.shape)

medium

A. (3, 16)

B. (3, 2, 16)

C. (2, 16)

D. (2, 3, 16)

Solution

1. Step 1: Understand input shape

   Input is a 2D tensor with shape (2, 3) representing 2 sequences each of length 3.

2. Step 2: Embedding output shape

   Embedding converts each integer to a 16-dimensional vector, so output shape is (2, 3, 16).

3. Final Answer:

   (2, 3, 16) -> Option D

4. Quick Check:

   Output shape = (batch_size, sequence_length, embedding_dim) [OK]

Hint: Output shape adds embedding dim to input shape [OK]

Common Mistakes:

• Mixing batch and sequence dimensions
• Forgetting embedding dimension in output
• Assuming output shape matches input shape exactly

4. Identify the error in the following embedding layer usage:

embedding = tf.keras.layers.Embedding(input_dim=1000, output_dim=64)
input_seq = tf.constant([[0, 1, 2], [999, 1000, 500]])
output = embedding(input_seq)

medium

A. The input sequence contains an index equal to input_dim, which is invalid

B. The output_dim is too large for the input_dim

C. Embedding layer requires input_dim and output_dim to be equal

D. The input sequence must be a list, not a tensor

Solution

1. Step 1: Check input indices validity

   Embedding indices must be in [0, input_dim-1]. Here, input_dim=1000, so max index is 999.

2. Step 2: Identify invalid index

   Input sequence contains 1000, which is out of range and causes an error.

3. Final Answer:

   The input sequence contains an index equal to input_dim, which is invalid -> Option A

4. Quick Check:

   Indices must be less than input_dim [OK]

Hint: Indices must be less than input_dim [OK]

Common Mistakes:

• Using index equal to input_dim
• Confusing output_dim size limits
• Thinking input must be list, not tensor

5. You want to use an embedding layer for a text classification task with a vocabulary of 10,000 words. You also want to limit the embedding size to 32 to reduce model size. Which approach is best to initialize the embedding layer?

hard

A. Use Embedding(input_dim=10000, output_dim=100) to get richer embeddings

B. Use Embedding(input_dim=10000, output_dim=32) with random initialization and train embeddings

C. Use one-hot encoding instead of embedding for smaller size

D. Use Embedding(input_dim=32, output_dim=10000) to reduce parameters

Solution

1. Step 1: Match embedding size to model constraints

   You want embedding size 32 to keep model small, so output_dim=32 is correct.

2. Step 2: Choose correct input_dim and initialization

   Input_dim must be vocabulary size 10,000. Random initialization is standard and embeddings are trained during model training.

3. Final Answer:

   Use Embedding(input_dim=10000, output_dim=32) with random initialization and train embeddings -> Option B

4. Quick Check:

   Embedding size = output_dim, vocab size = input_dim [OK]

Hint: Match input_dim to vocab, output_dim to embedding size [OK]

Common Mistakes:

• Swapping input_dim and output_dim
• Using one-hot encoding for large vocab
• Choosing embedding size too large for constraints