Limitations of classical methods in NLP - ML Experiment: Train & Evaluate

Experiment - Limitations of classical methods

Problem:Classical NLP methods like bag-of-words and TF-IDF are used to classify movie reviews as positive or negative. The current model uses a simple logistic regression on TF-IDF features.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model overfits the training data and performs poorly on validation data, showing classical methods struggle with capturing context and semantics.

Your Task

Reduce overfitting and improve validation accuracy to at least 80% while keeping training accuracy below 90%.

Keep using classical feature extraction methods (TF-IDF or bag-of-words).

Do not use deep learning or pretrained embeddings.

You can adjust model hyperparameters and add regularization.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, log_loss

# Load movie review data (subset of 20 newsgroups for example)
data = fetch_20newsgroups(subset='all', categories=['rec.autos', 'rec.sport.baseball'], remove=('headers', 'footers', 'quotes'))
X_train, X_val, y_train, y_val = train_test_split(data.data, data.target, test_size=0.2, random_state=42)

# TF-IDF vectorizer with limited features and bigrams
vectorizer = TfidfVectorizer(max_features=5000, ngram_range=(1,2))
X_train_tfidf = vectorizer.fit_transform(X_train)
X_val_tfidf = vectorizer.transform(X_val)

# Logistic Regression with L2 regularization and tuned C
model = LogisticRegression(penalty='l2', C=0.5, max_iter=200, random_state=42)
model.fit(X_train_tfidf, y_train)

# Predictions and metrics
train_preds = model.predict(X_train_tfidf)
val_preds = model.predict(X_val_tfidf)
train_probs = model.predict_proba(X_train_tfidf)
val_probs = model.predict_proba(X_val_tfidf)

train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100
train_loss = log_loss(y_train, train_probs)
val_loss = log_loss(y_val, val_probs)

print(f"Training accuracy: {train_acc:.2f}%")
print(f"Validation accuracy: {val_acc:.2f}%")
print(f"Training loss: {train_loss:.2f}")
print(f"Validation loss: {val_loss:.2f}")

Limited TF-IDF features to 5000 to reduce noise and overfitting.

Added bigrams to capture some word context.

Applied L2 regularization in logistic regression with C=0.5 to reduce overfitting.

Increased max_iter to ensure convergence.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.60

After: Training accuracy 88%, Validation accuracy 82%, Training loss 0.30, Validation loss 0.45

Classical methods like TF-IDF with logistic regression can overfit easily due to high feature dimensionality and lack of semantic understanding. Adding regularization and limiting features helps reduce overfitting and improves validation accuracy, but classical methods still struggle to capture deep context compared to modern approaches.

Bonus Experiment

Try using n-grams up to trigrams and compare validation accuracy and overfitting.

💡 Hint

Increasing n-gram range can capture more context but may increase feature size and overfitting risk. Adjust max_features and regularization accordingly.

Practice

1. Which of the following is a main limitation of classical NLP methods like bag-of-words?

easy

A. They ignore the order and context of words in a sentence.

B. They require very large datasets to work.

C. They always need deep neural networks to function.

D. They can understand sarcasm and irony easily.

Solution

1. Step 1: Understand classical NLP methods

   Classical methods like bag-of-words treat text as a collection of words without order or context.

2. Step 2: Identify the limitation

   This means they cannot capture meaning that depends on word order or surrounding words.

3. Final Answer:

   They ignore the order and context of words in a sentence. -> Option A

4. Quick Check:

   Classical methods miss context = C [OK]

Hint: Remember bag-of-words loses word order and context [OK]

Common Mistakes:

• Thinking classical methods need big data
• Believing classical methods use deep learning
• Assuming classical methods understand sarcasm

2. Which syntax correctly represents a classical method feature extraction for text using bag-of-words in Python?

easy

A. import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp(text)

B. import tensorflow as tf
model = tf.keras.Sequential()

C. from nltk.tokenize import word_tokenize
words = word_tokenize(text)

D. from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

Solution

1. Step 1: Identify classical method for feature extraction

   Bag-of-words uses CountVectorizer from sklearn to convert text to word counts.

2. Step 2: Match syntax to bag-of-words

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() X = vectorizer.fit_transform(texts) shows correct import and usage of CountVectorizer for feature extraction.

3. Final Answer:

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() X = vectorizer.fit_transform(texts) -> Option D

4. Quick Check:

   CountVectorizer syntax = A [OK]

Hint: CountVectorizer is sklearn's bag-of-words tool [OK]

Common Mistakes:

• Confusing tokenization with feature extraction
• Using deep learning imports for classical methods
• Mixing spaCy usage with bag-of-words

3. Given this code using bag-of-words, what is the shape of the output matrix X if texts = ['I love AI', 'love AI']?

medium

A. (2, 4)

B. (3, 2)

C. (2, 3)

D. (4, 2)

Solution

1. Step 1: Count unique words in texts

   Texts are ['I love AI', 'love AI']. Lowercased tokens: 'i love ai', 'love ai'. Unique tokens: 'ai', 'i', 'love' = 3 words.

2. Step 2: Check CountVectorizer default behavior

   CountVectorizer lowercases and tokenizes. Number of samples is 2. So shape is (2, 3).

3. Final Answer:

   (2, 3) -> Option C

4. Quick Check:

   2 samples, 3 features = B [OK]

Hint: Count unique words for shape: rows=samples, cols=unique words [OK]

Common Mistakes:

• Counting words instead of unique tokens
• Mixing rows and columns in shape
• Ignoring case sensitivity

4. Identify the error in this classical NLP code snippet using CountVectorizer:

from sklearn.feature_extraction.text import CountVectorizer
texts = ['Hello world', 'Hello']
vectorizer = CountVectorizer()
X = vectorizer.fit(texts)
print(X.toarray())

medium

A. fit() should be fit_transform() to get the matrix.

B. CountVectorizer cannot process lists of strings.

C. toarray() is not a method of the output.

D. Missing import for numpy.

Solution

1. Step 1: Check CountVectorizer usage

   fit() learns the vocabulary but does not transform texts to matrix. fit_transform() does both.

2. Step 2: Identify correct method to get matrix

   To get the document-term matrix, fit_transform() must be used. Using fit() alone returns the vectorizer object, which has no toarray() method.

3. Final Answer:

   fit() should be fit_transform() to get the matrix. -> Option A

4. Quick Check:

   fit_transform() needed for matrix [OK]

Hint: Use fit_transform() to get matrix, not just fit() [OK]

Common Mistakes:

• Using fit() instead of fit_transform()
• Assuming toarray() works on vectorizer
• Thinking CountVectorizer needs numpy import

5. Why might classical NLP methods like bag-of-words fail on sentiment analysis of complex sentences such as 'I don't think this movie was good'?

hard

A. They cannot tokenize contractions like "don't".

B. They treat words independently and miss negation and word order.

C. They always overfit on small datasets.

D. They require GPU acceleration to process negations.

Solution

1. Step 1: Understand classical method limitations

   Bag-of-words treats each word separately, ignoring order and context.

2. Step 2: Analyze sentence complexity

   Sentence has negation "don't" which flips sentiment. Without context, model may misinterpret sentiment.

3. Step 3: Identify why classical methods fail

   Because they ignore word order and negation, they fail to capture true sentiment.

4. Final Answer:

   They treat words independently and miss negation and word order. -> Option B

5. Quick Check:

   Miss negation and order = D [OK]

Hint: Negation needs context; classical methods miss it [OK]

Common Mistakes:

• Thinking classical methods need GPUs
• Believing classical methods can't tokenize contractions
• Confusing overfitting with context loss