Why TensorFlow handles text effectively - Experiment to Prove It

Experiment - Why TensorFlow handles text effectively

Problem:You want to build a model that understands text data, like movie reviews, to predict if the review is positive or negative.

Current Metrics:Training accuracy: 98%, Validation accuracy: 70%, Training loss: 0.05, Validation loss: 0.85

Issue:The model is overfitting. It learns the training data too well but does not generalize to new text data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You must use TensorFlow and its text processing tools.

You can only change model architecture and training parameters, not the dataset.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Sample dataset: IMDB reviews (already split)
(train_data, train_labels), (test_data, test_labels) = tf.keras.datasets.imdb.load_data(num_words=10000)

# Decode integers back to words for TextVectorization demonstration
word_index = tf.keras.datasets.imdb.get_word_index()
index_word = {v + 3: k for k, v in word_index.items()}
index_word[0] = '<PAD>'
index_word[1] = '<START>'
index_word[2] = '<UNK>'
index_word[3] = '<UNUSED>'

# Convert integer sequences back to text
train_text = [' '.join([index_word.get(i, '?') for i in seq]) for seq in train_data]

# Create TextVectorization layer
max_features = 10000
sequence_length = 250
vectorize_layer = layers.TextVectorization(
    max_tokens=max_features,
    output_mode='int',
    output_sequence_length=sequence_length
)

# Adapt vectorization layer to training text
vectorize_layer.adapt(train_text)

# Build model
model = models.Sequential([
    vectorize_layer,
    layers.Embedding(max_features + 1, 64),
    layers.Dropout(0.5),
    layers.GlobalAveragePooling1D(),
    layers.Dropout(0.5),
    layers.Dense(64, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])

model.compile(
    loss='binary_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

# Prepare validation data
val_text = [' '.join([index_word.get(i, '?') for i in seq]) for seq in test_data]

# Early stopping callback
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# Train model
history = model.fit(
    train_text, train_labels,
    epochs=20,
    batch_size=512,
    validation_data=(val_text, test_labels),
    callbacks=[early_stop],
    verbose=2
)

# Output final metrics
final_train_acc = history.history['accuracy'][-1] * 100
final_val_acc = history.history['val_accuracy'][-1] * 100
final_train_loss = history.history['loss'][-1]
final_val_loss = history.history['val_loss'][-1]

print(f'Training accuracy: {final_train_acc:.2f}%')
print(f'Validation accuracy: {final_val_acc:.2f}%')
print(f'Training loss: {final_train_loss:.4f}')
print(f'Validation loss: {final_val_loss:.4f}')

Used TensorFlow's TextVectorization layer to convert raw text into integer sequences efficiently.

Added an Embedding layer to represent words as vectors capturing meaning.

Inserted Dropout layers to reduce overfitting by randomly ignoring some neurons during training.

Used GlobalAveragePooling1D to summarize sequence information simply.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy was 98%, validation accuracy was 70%, showing strong overfitting.

After: Training accuracy dropped to 90%, validation accuracy improved to 87%, showing better generalization.

TensorFlow handles text effectively by providing tools like TextVectorization and Embedding layers that simplify text preprocessing and representation. Adding dropout and early stopping helps reduce overfitting, improving model performance on new text data.

Bonus Experiment

Try replacing the GlobalAveragePooling1D layer with an LSTM layer to capture word order better and see if validation accuracy improves.

💡 Hint

Use layers.LSTM(64) after the Embedding layer and before Dropout.

Practice

1. Why does TensorFlow convert text into numbers before processing?

easy

A. Because numbers are easier to print

B. Because text files are too large to handle

C. Because computers understand numbers better than text

D. Because text cannot be stored in memory

Solution

1. Step 1: Understand computer data processing

   Computers work with numbers internally, not raw text characters.

2. Step 2: Reason why text is converted

   Converting text to numbers allows TensorFlow to perform math operations needed for learning.

3. Final Answer:

   Because computers understand numbers better than text -> Option C

4. Quick Check:

   Text to numbers = computers understand numbers [OK]

Hint: Remember: computers only understand numbers, not words [OK]

Common Mistakes:

• Thinking text files are too large to process
• Believing numbers are easier to print than text
• Assuming text cannot be stored in memory

2. Which TensorFlow layer is commonly used to turn words into vectors?

easy

A. Dense layer

B. Embedding layer

C. Dropout layer

D. Conv2D layer

Solution

1. Step 1: Identify layers for text processing

   Embedding layers convert words into numeric vectors capturing meaning.

2. Step 2: Compare with other layers

   Dense layers are for general neural nets, Dropout prevents overfitting, Conv2D is for images.

3. Final Answer:

   Embedding layer -> Option B

4. Quick Check:

   Word to vector = Embedding layer [OK]

Hint: Embedding layers map words to numbers smartly [OK]

Common Mistakes:

• Choosing Dense layer for word vectors
• Confusing Conv2D with text layers
• Thinking Dropout creates vectors

3. What will be the output shape of this TensorFlow code?

import tensorflow as tf
layer = tf.keras.layers.Embedding(input_dim=1000, output_dim=64)
input_data = tf.constant([[1, 2, 3], [4, 5, 6]])
output = layer(input_data)
print(output.shape)

medium

A. (2, 3, 64)

B. (3, 2, 64)

C. (2, 64)

D. (3, 64)

Solution

1. Step 1: Understand input shape

   Input is a 2x3 tensor (2 rows, 3 words each).

2. Step 2: Embedding output shape

   Embedding converts each word index to a 64-length vector, so output shape is (2, 3, 64).

3. Final Answer:

   (2, 3, 64) -> Option A

4. Quick Check:

   Input shape (2,3) + vector size 64 = (2,3,64) [OK]

Hint: Embedding output shape = input shape + vector size [OK]

Common Mistakes:

• Swapping batch and sequence dimensions
• Forgetting embedding vector size
• Assuming output is 2D instead of 3D

4. Identify the error in this TensorFlow text processing code:

import tensorflow as tf
layer = tf.keras.layers.Embedding(input_dim=5000, output_dim=32)
input_data = tf.constant([1, 2, 3, 4])
output = layer(input_data)
print(output.shape)

medium

A. input_data should be 2D, not 1D

B. output_dim must be larger than input_dim

C. Embedding layer requires float input

D. Missing activation function in Embedding

Solution

1. Step 1: Check input shape requirements

   Embedding expects 2D input: (batch_size, sequence_length), but input_data is 1D.

2. Step 2: Confirm other parameters

   output_dim can be smaller than input_dim; input can be integer indices; activation not required.

3. Final Answer:

   input_data should be 2D, not 1D -> Option A

4. Quick Check:

   Embedding input must be 2D tensor [OK]

Hint: Embedding input must be 2D (batch, sequence) [OK]

Common Mistakes:

• Using 1D input instead of 2D
• Thinking output_dim must exceed input_dim
• Adding activation to Embedding unnecessarily

5. You want to build a chatbot using TensorFlow that understands word meaning and context. Which approach best helps TensorFlow handle text effectively?

hard

A. Convert text to uppercase and feed directly to a Conv2D layer

B. Use only a Dense layer on raw text input

C. Use Dropout layers only to prevent overfitting

D. Use an Embedding layer followed by an LSTM layer to capture word vectors and sequence context

Solution

1. Step 1: Understand text representation needs

   Embedding layers convert words to vectors capturing meaning; LSTM layers capture sequence context.

2. Step 2: Evaluate other options

   Dense layers alone can't handle sequences; Conv2D is for images; Dropout only prevents overfitting, not understanding.

3. Final Answer:

   Use an Embedding layer followed by an LSTM layer to capture word vectors and sequence context -> Option D

4. Quick Check:

   Embedding + LSTM = best for text understanding [OK]

Hint: Combine Embedding and LSTM for smart text apps [OK]

Common Mistakes:

• Using Dense layers on raw text
• Applying Conv2D to text data
• Relying only on Dropout layers