Limitations of classical methods in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why classical ML methods struggle with high-dimensional text data

Classical machine learning methods like Naive Bayes or SVM often face challenges when working with text data represented by thousands of features (words). What is the main reason for this difficulty?

A They do not support categorical features like words

B They cannot handle numeric data and text is always numeric

C They always produce biased predictions regardless of data size

D They require very large amounts of labeled data to avoid overfitting in high dimensions

💡 Hint

Think about what happens when the number of features is much larger than the number of training examples.

Classical methods often overfit when the feature space is very large compared to the number of samples. This is common in text data where each word is a feature, leading to thousands of features but limited labeled examples.

❓ Model Choice

intermediate

Choosing a model to handle complex language patterns

Which classical machine learning model is least suitable for capturing complex word order and context in sentences?

A Logistic Regression with n-gram features

B Naive Bayes with bag-of-words features

C Recurrent Neural Network (RNN)

D Support Vector Machine with TF-IDF features

💡 Hint

Consider which model assumes independence between words and ignores order.

Naive Bayes assumes word independence and uses bag-of-words, so it cannot capture word order or context well. Other models like RNNs or n-gram features consider some order or context.

❓ Metrics

advanced

Evaluating classical methods on imbalanced text data

You train a classical classifier on a text dataset where 95% of examples belong to one class. The model achieves 95% accuracy but poor recall on the minority class. What metric better reflects the model's weakness?

A Recall on the minority class

B Overall accuracy

C Precision on the minority class

D Training loss

💡 Hint

Which metric measures how many true positive minority examples are correctly found?

Recall on the minority class measures the fraction of actual minority examples correctly identified. High accuracy can be misleading if the model ignores the minority class.

🔧 Debug

advanced

Why does this classical text classifier fail to generalize?

Consider this Python snippet using a classical method for text classification:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ["good movie", "bad movie", "great film", "terrible film"]
labels = [1, 0, 1, 0]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(X, labels)

new_text = ["good film"]
X_new = vectorizer.transform(new_text)
pred = model.predict(X_new)
print(pred)

Why might this model give unreliable predictions on new text?

NLP

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ["good movie", "bad movie", "great film", "terrible film"]
labels = [1, 0, 1, 0]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(X, labels)

new_text = ["good film"]
X_new = vectorizer.transform(new_text)
pred = model.predict(X_new)
print(pred)

A The training data is too small and limited to learn reliable word associations

B CountVectorizer does not convert text to numbers

C MultinomialNB cannot handle sparse matrices

D The new text contains words not seen during training

💡 Hint

Think about the size and variety of the training examples.

The training set is very small with only four examples, so the model cannot learn strong patterns. This causes unreliable predictions on new inputs even if the words are known.

🧠 Conceptual

expert

Fundamental limitation of classical methods in capturing semantics

Classical machine learning methods for NLP often rely on word frequency features like bag-of-words or TF-IDF. What is a fundamental limitation of these features in understanding language?

A They require deep neural networks to compute

B They always produce dense vectors that are hard to interpret

C They ignore word order and context, so cannot capture meaning beyond individual words

D They cannot be used with linear classifiers

💡 Hint

Think about what meaning depends on besides word counts.

Bag-of-words and TF-IDF count words but ignore their order and surrounding words, so they miss context and semantic relationships essential for true language understanding.

Practice

1. Which of the following is a main limitation of classical NLP methods like bag-of-words?

easy

A. They ignore the order and context of words in a sentence.

B. They require very large datasets to work.

C. They always need deep neural networks to function.

D. They can understand sarcasm and irony easily.

Solution

1. Step 1: Understand classical NLP methods

   Classical methods like bag-of-words treat text as a collection of words without order or context.

2. Step 2: Identify the limitation

   This means they cannot capture meaning that depends on word order or surrounding words.

3. Final Answer:

   They ignore the order and context of words in a sentence. -> Option A

4. Quick Check:

   Classical methods miss context = C [OK]

Hint: Remember bag-of-words loses word order and context [OK]

Common Mistakes:

• Thinking classical methods need big data
• Believing classical methods use deep learning
• Assuming classical methods understand sarcasm

2. Which syntax correctly represents a classical method feature extraction for text using bag-of-words in Python?

easy

A. import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp(text)

B. import tensorflow as tf
model = tf.keras.Sequential()

C. from nltk.tokenize import word_tokenize
words = word_tokenize(text)

D. from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

Solution

1. Step 1: Identify classical method for feature extraction

   Bag-of-words uses CountVectorizer from sklearn to convert text to word counts.

2. Step 2: Match syntax to bag-of-words

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() X = vectorizer.fit_transform(texts) shows correct import and usage of CountVectorizer for feature extraction.

3. Final Answer:

   from sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() X = vectorizer.fit_transform(texts) -> Option D

4. Quick Check:

   CountVectorizer syntax = A [OK]

Hint: CountVectorizer is sklearn's bag-of-words tool [OK]

Common Mistakes:

• Confusing tokenization with feature extraction
• Using deep learning imports for classical methods
• Mixing spaCy usage with bag-of-words

3. Given this code using bag-of-words, what is the shape of the output matrix X if texts = ['I love AI', 'love AI']?

medium

A. (2, 4)

B. (3, 2)

C. (2, 3)

D. (4, 2)

Solution

1. Step 1: Count unique words in texts

   Texts are ['I love AI', 'love AI']. Lowercased tokens: 'i love ai', 'love ai'. Unique tokens: 'ai', 'i', 'love' = 3 words.

2. Step 2: Check CountVectorizer default behavior

   CountVectorizer lowercases and tokenizes. Number of samples is 2. So shape is (2, 3).

3. Final Answer:

   (2, 3) -> Option C

4. Quick Check:

   2 samples, 3 features = B [OK]

Hint: Count unique words for shape: rows=samples, cols=unique words [OK]

Common Mistakes:

• Counting words instead of unique tokens
• Mixing rows and columns in shape
• Ignoring case sensitivity

4. Identify the error in this classical NLP code snippet using CountVectorizer:

from sklearn.feature_extraction.text import CountVectorizer
texts = ['Hello world', 'Hello']
vectorizer = CountVectorizer()
X = vectorizer.fit(texts)
print(X.toarray())

medium

A. fit() should be fit_transform() to get the matrix.

B. CountVectorizer cannot process lists of strings.

C. toarray() is not a method of the output.

D. Missing import for numpy.

Solution

1. Step 1: Check CountVectorizer usage

   fit() learns the vocabulary but does not transform texts to matrix. fit_transform() does both.

2. Step 2: Identify correct method to get matrix

   To get the document-term matrix, fit_transform() must be used. Using fit() alone returns the vectorizer object, which has no toarray() method.

3. Final Answer:

   fit() should be fit_transform() to get the matrix. -> Option A

4. Quick Check:

   fit_transform() needed for matrix [OK]

Hint: Use fit_transform() to get matrix, not just fit() [OK]

Common Mistakes:

• Using fit() instead of fit_transform()
• Assuming toarray() works on vectorizer
• Thinking CountVectorizer needs numpy import

5. Why might classical NLP methods like bag-of-words fail on sentiment analysis of complex sentences such as 'I don't think this movie was good'?

hard

A. They cannot tokenize contractions like "don't".

B. They treat words independently and miss negation and word order.

C. They always overfit on small datasets.

D. They require GPU acceleration to process negations.

Solution

1. Step 1: Understand classical method limitations

   Bag-of-words treats each word separately, ignoring order and context.

2. Step 2: Analyze sentence complexity

   Sentence has negation "don't" which flips sentiment. Without context, model may misinterpret sentiment.

3. Step 3: Identify why classical methods fail

   Because they ignore word order and negation, they fail to capture true sentiment.

4. Final Answer:

   They treat words independently and miss negation and word order. -> Option B

5. Quick Check:

   Miss negation and order = D [OK]

Hint: Negation needs context; classical methods miss it [OK]

Common Mistakes:

• Thinking classical methods need GPUs
• Believing classical methods can't tokenize contractions
• Confusing overfitting with context loss