Embedding layer usage in NLP

Introduction

An embedding layer helps turn words or items into numbers that a computer can understand better. It makes learning from text or categories easier and smarter.

When you want to teach a computer to understand words in a sentence.

When you have categories or labels and want to represent them as numbers.

When you want to reduce the size of data by turning many words into smaller number groups.

When building chatbots or language translation tools.

When working with any data that has discrete items like words, products, or users.

Syntax

NLP

Embedding(input_dim, output_dim, input_length=None)

input_dim: Number of unique items (like words) you have.

output_dim: Size of the number group (vector) each item will become.

Examples

This creates an embedding for 1000 unique words, each represented by 64 numbers.

NLP

Embedding(1000, 64)

This creates embeddings for 5000 words, each with 128 numbers, expecting input sequences of length 10.

NLP

Embedding(5000, 128, input_length=10)

Sample Model

This example shows how to use an embedding layer to turn word indices into vectors. The model learns to predict a label from sequences of words.

NLP

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Flatten, Dense
import numpy as np

# Suppose we have 10 unique words (0 to 9)
# Each sentence has 4 words
vocab_size = 10
embedding_dim = 8
input_length = 4

# Create a simple model with an embedding layer
model = Sequential([
    Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=input_length),
    Flatten(),  # Flatten the 2D embeddings to 1D
    Dense(1, activation='sigmoid')  # Output layer for binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Sample input: batch of 2 sentences, each with 4 words (word indices)
x_train = np.array([[1, 2, 3, 4], [4, 3, 2, 1]])
y_train = np.array([0, 1])  # Sample labels

# Train the model for 3 epochs
history = model.fit(x_train, y_train, epochs=3, verbose=0)

# Predict on new data
x_test = np.array([[1, 2, 3, 4]])
predictions = model.predict(x_test)

print(f"Predictions: {predictions.flatten()}")
print(f"Training accuracy after 3 epochs: {history.history['accuracy'][-1]:.4f}")

Important Notes

The embedding layer learns the best number representations during training.

Input to the embedding layer must be integers representing words or categories.

Embedding output shape is (batch_size, input_length, output_dim).

Summary

Embedding layers turn words or categories into useful numbers for models.

They help models understand relationships between items.

Use embeddings when working with text or categorical data.