Text classification model in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Text classification model

Problem:Classify movie reviews as positive or negative using a neural network.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.65

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only change the model architecture and training parameters.

Do not change the dataset or preprocessing steps.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.callbacks import EarlyStopping

# Load dataset
(train_data, train_labels), (test_data, test_labels) = tf.keras.datasets.imdb.load_data(num_words=10000)

# Pad sequences to the same length
train_data = tf.keras.preprocessing.sequence.pad_sequences(train_data, maxlen=256)
test_data = tf.keras.preprocessing.sequence.pad_sequences(test_data, maxlen=256)

# Build model with dropout and smaller dense layer
model = models.Sequential([
    layers.Embedding(10000, 16, input_length=256),
    layers.GlobalAveragePooling1D(),
    layers.Dropout(0.5),
    layers.Dense(16, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    layers.Dropout(0.5),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Use early stopping to prevent overfitting
early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(train_data, train_labels,
                    epochs=30,
                    batch_size=512,
                    validation_split=0.2,
                    callbacks=[early_stop],
                    verbose=2)

# Evaluate on test data
results = model.evaluate(test_data, test_labels, verbose=2)

print(f"Test Loss: {results[0]:.4f}, Test Accuracy: {results[1]*100:.2f}%")

Added dropout layers with 50% rate after embedding and dense layers to reduce overfitting.

Reduced dense layer units from 64 to 16 to simplify the model.

Added L2 regularization to dense layer weights.

Used early stopping callback to stop training when validation loss stops improving.

Lowered learning rate to 0.001 for more stable training.

Results Interpretation

Before: Training accuracy was 98%, validation accuracy was 75%. The model was memorizing training data but not generalizing well.

After: Training accuracy dropped to 90%, validation accuracy improved to 87%. The model now generalizes better to new data.

Adding dropout, regularization, and early stopping helps reduce overfitting. This improves validation accuracy by preventing the model from memorizing training data.

Bonus Experiment

Try using a convolutional neural network (CNN) instead of a simple dense model for text classification.

💡 Hint

Use Conv1D and MaxPooling1D layers after the embedding layer to capture local word patterns.

Practice

1. What is the main purpose of a text classification model in TensorFlow?

easy

A. To sort text into different categories based on content

B. To generate new text from a given input

C. To translate text from one language to another

D. To detect objects in images

Solution

1. Step 1: Understand the task of text classification

   Text classification means assigning labels or categories to text data based on its content.

2. Step 2: Match the task to the options

   To sort text into different categories based on content describes sorting text into categories, which matches text classification.

3. Final Answer:

   To sort text into different categories based on content -> Option A

4. Quick Check:

   Text classification = sorting text [OK]

Hint: Text classification means sorting text into groups [OK]

Common Mistakes:

• Confusing classification with text generation
• Thinking classification is for images
• Mixing up translation with classification

2. Which of the following is the correct way to add an embedding layer in a TensorFlow Sequential model for text classification?

easy

A. model.add(tf.keras.layers.Dense(64, input_dim=1000))

B. model.add(tf.keras.layers.Conv2D(64, kernel_size=3))

C. model.add(tf.keras.layers.Embedding(input_dim=1000, output_dim=64))

D. model.add(tf.keras.layers.LSTM(64, input_shape=(1000,)))

Solution

1. Step 1: Identify the embedding layer syntax

   The embedding layer in TensorFlow is added with tf.keras.layers.Embedding and requires input_dim and output_dim parameters.

2. Step 2: Check each option

   model.add(tf.keras.layers.Embedding(input_dim=1000, output_dim=64)) correctly uses Embedding with input_dim=1000 and output_dim=64. Others use Dense, Conv2D, or LSTM which are not embedding layers.

3. Final Answer:

   model.add(tf.keras.layers.Embedding(input_dim=1000, output_dim=64)) -> Option C

4. Quick Check:

   Embedding layer uses tf.keras.layers.Embedding [OK]

Hint: Embedding layer uses tf.keras.layers.Embedding with input/output dims [OK]

Common Mistakes:

• Using Dense or Conv2D instead of Embedding
• Missing input_dim or output_dim parameters
• Confusing LSTM with embedding layer

3. Given the following TensorFlow model code for text classification, what will be the shape of the output layer's predictions for a batch of 32 samples if there are 5 classes?

model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=1000, output_dim=16, input_length=10),
  tf.keras.layers.GlobalAveragePooling1D(),
  tf.keras.layers.Dense(5, activation='softmax')
])

predictions = model(tf.random.uniform((32, 10), maxval=1000, dtype=tf.int32))

medium

A. (10, 5)

B. (32, 5)

C. (32, 10)

D. (5, 32)

Solution

1. Step 1: Analyze model output layer

   The last Dense layer has 5 units with softmax, so output shape per sample is (5,).

2. Step 2: Consider batch size

   Input batch size is 32, so output shape is (32, 5).

3. Final Answer:

   (32, 5) -> Option B

4. Quick Check:

   Batch size 32 x 5 classes = (32, 5) [OK]

Hint: Output shape = (batch size, number of classes) [OK]

Common Mistakes:

• Confusing sequence length with batch size
• Mixing up input and output shapes
• Ignoring batch dimension in output

4. You wrote this TensorFlow model for text classification but get a shape error during training:

model = tf.keras.Sequential([
  tf.keras.layers.Embedding(input_dim=5000, output_dim=32),
  tf.keras.layers.Dense(10, activation='relu'),
  tf.keras.layers.Dense(3, activation='softmax')
])

What is the likely cause of the error?

medium

A. Dense layers cannot follow Embedding layers

B. Using wrong activation function in the last layer

C. input_dim in Embedding is too small

D. Missing a Flatten or GlobalAveragePooling1D layer after Embedding

Solution

1. Step 1: Understand Embedding output shape

   Embedding outputs 3D tensor (batch, sequence_length, embedding_dim). Dense expects 2D input.

2. Step 2: Identify missing layer

   Without Flatten or GlobalAveragePooling1D, Dense layer gets wrong input shape causing error.

3. Final Answer:

   Missing a Flatten or GlobalAveragePooling1D layer after Embedding -> Option D

4. Quick Check:

   Embedding output must be flattened before Dense [OK]

Hint: Add pooling or flatten after Embedding before Dense layers [OK]

Common Mistakes:

• Assuming Dense accepts 3D input
• Changing activation won't fix shape error
• Believing Dense can't follow Embedding at all

5. You want to build a text classification model in TensorFlow that handles variable-length input sentences and classifies them into 4 categories. Which model design is best?

hard

A. Use Embedding layer, GlobalAveragePooling1D, then Dense with 4 units and softmax activation

B. Use Embedding layer, Flatten, then Dense with 4 units and sigmoid activation

C. Use Embedding layer, Conv2D, then Dense with 1 unit and softmax activation

D. Use Dense layers only without Embedding

Solution

1. Step 1: Handle variable-length input

   GlobalAveragePooling1D can convert variable-length sequences to fixed-size vectors.

2. Step 2: Choose output layer for multi-class classification

   Dense with 4 units and softmax activation outputs probabilities for 4 classes.

3. Final Answer:

   Use Embedding layer, GlobalAveragePooling1D, then Dense with 4 units and softmax activation -> Option A

4. Quick Check:

   Pooling + softmax = best for variable-length multi-class [OK]

Hint: Use pooling + softmax for variable-length multi-class text [OK]

Common Mistakes:

• Using Flatten breaks variable-length input handling
• Using sigmoid for multi-class instead of softmax
• Using Conv2D for text sequences incorrectly