Why text classification categorizes documents in NLP - Experiment to Prove It

Experiment - Why text classification categorizes documents

Problem:We want to teach a computer to read short text messages and decide what category they belong to, like 'sports', 'technology', or 'health'. Currently, the model guesses correctly 95% of the time on the training messages but only 70% on new messages it hasn't seen before.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting. It learns the training messages too well but does not generalize to new messages.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change the model architecture and training settings.

Do not add more data or change the dataset.

Keep the text preprocessing steps the same.

Hint 1

Hint 2

Hint 3

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, GlobalAveragePooling1D, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preprocessed and ready

vocab_size = 10000
embedding_dim = 16
max_length = 100

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_length),
    GlobalAveragePooling1D(),
    Dropout(0.5),
    Dense(16, activation='relu'),
    Dropout(0.5),
    Dense(3, activation='softmax')  # 3 categories
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(
    X_train, y_train,
    epochs=30,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop]
)

Added Dropout layers with 50% rate after embedding and dense layers to reduce overfitting.

Reduced the dense layer size from a larger number to 16 neurons to simplify the model.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy was 95%, validation accuracy was 70%, showing overfitting.

After: Training accuracy dropped to 90%, validation accuracy improved to 87%, showing better generalization.

Adding dropout and early stopping helps the model not memorize training data too much, so it performs better on new unseen texts.

Bonus Experiment

Try using a simpler model like logistic regression with TF-IDF features instead of a neural network.

💡 Hint

Use scikit-learn's TfidfVectorizer and LogisticRegression to see if a simpler model can also reduce overfitting.

Practice

1. Why do we use text classification in organizing documents?

easy

A. To automatically group documents by their content

B. To delete documents that are not useful

C. To translate documents into different languages

D. To create new documents from existing ones

Solution

1. Step 1: Understand the purpose of text classification

   Text classification is used to sort or group documents based on what they talk about.

2. Step 2: Identify the correct use case

   Among the options, only grouping documents by content matches the purpose of text classification.

3. Final Answer:

   To automatically group documents by their content -> Option A

4. Quick Check:

   Text classification = grouping documents [OK]

Hint: Text classification groups by content, not deletes or translates [OK]

Common Mistakes:

• Confusing classification with translation
• Thinking classification deletes documents
• Assuming classification creates new documents

2. Which of the following is the correct way to describe text classification?

easy

A. It removes stop words from text

B. It translates text into numbers for storage

C. It assigns labels to text based on content

D. It generates new text from existing text

Solution

1. Step 1: Define text classification

   Text classification means giving a label or category to a piece of text based on what it contains.

2. Step 2: Match the definition to options

   Only assigning labels based on content matches the definition of text classification.

3. Final Answer:

   It assigns labels to text based on content -> Option C

4. Quick Check:

   Assign labels = classification [OK]

Hint: Classification means labeling, not translating or generating [OK]

Common Mistakes:

• Mixing classification with text preprocessing
• Confusing classification with text generation
• Thinking classification is about data storage

3. Given this Python code snippet for text classification, what will be the output?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ['I love cats', 'I hate rain', 'Cats are great', 'Rain is bad']
labels = ['positive', 'negative', 'positive', 'negative']

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(X, labels)

new_text = ['I love rain']
X_new = vectorizer.transform(new_text)
prediction = model.predict(X_new)
print(prediction[0])

medium

A. negative

B. positive

C. neutral

D. error

Solution

1. Step 1: Understand training data and labels

   The model learns 'I love cats' and 'Cats are great' as positive, 'I hate rain' and 'Rain is bad' as negative.

2. Step 2: Predict label for 'I love rain'

   The word 'love' appears in positive examples, and 'rain' appears in negative examples. The model weighs 'love' more strongly positive, so prediction is 'positive'.

3. Final Answer:

   positive -> Option B

4. Quick Check:

   Model predicts 'positive' for 'I love rain' [OK]

Hint: Words linked to positive examples influence prediction [OK]

Common Mistakes:

• Assuming 'love' always makes prediction positive
• Ignoring word frequency impact
• Expecting neutral label which is not in training

4. Find the error in this text classification code snippet:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ['happy day', 'sad night']
labels = ['positive', 'negative']

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(texts, labels)  # Error here

new_text = ['happy night']
X_new = vectorizer.transform(new_text)
prediction = model.predict(X_new)
print(prediction[0])

medium

A. Transforming new_text before vectorizing

B. Missing import for MultinomialNB

C. Labels list is empty

D. Using texts instead of X in model.fit

Solution

1. Step 1: Check model.fit inputs

   Model expects numeric features (X), but texts (strings) are passed instead.

2. Step 2: Correct the input to model.fit

   Replace texts with X (vectorized data) to fix the error.

3. Final Answer:

   Using texts instead of X in model.fit -> Option D

4. Quick Check:

   model.fit needs numeric input X [OK]

Hint: Model.fit needs vectorized data, not raw text [OK]

Common Mistakes:

• Passing raw text instead of vectorized features
• Ignoring error messages about input types
• Confusing transform and fit_transform

5. You want to classify news articles into categories like 'sports', 'politics', and 'technology'. Which approach best explains why text classification helps here?

hard

A. It learns patterns from labeled articles to predict categories for new articles

B. It translates articles into multiple languages for wider reach

C. It summarizes articles to reduce reading time

D. It deletes irrelevant articles automatically

Solution

1. Step 1: Understand the goal of classifying news articles

   The goal is to assign correct categories to new articles based on past examples.

2. Step 2: Identify how text classification achieves this

   Text classification learns from labeled data patterns to predict categories for unseen articles.

3. Final Answer:

   It learns patterns from labeled articles to predict categories for new articles -> Option A

4. Quick Check:

   Learning from examples = classification [OK]

Hint: Classification learns from examples to label new data [OK]

Common Mistakes:

• Confusing classification with translation or summarization
• Thinking classification deletes data
• Assuming classification creates content