Why text classification categorizes documents in NLP - Why Metrics Matter

Metrics & Evaluation - Why text classification categorizes documents

Which metric matters for this concept and WHY

For text classification, accuracy shows overall correct predictions. But because some categories may be rare, precision and recall are very important.

Precision tells us how many documents labeled as a category truly belong there. This avoids false alarms.

Recall tells us how many documents of a category were found by the model. This avoids missing important documents.

F1 score balances precision and recall, giving a single number to compare models.

Confusion matrix example

      Actual \ Predicted | Sports | Politics | Tech | Total
      ---------------------------------------------------
      Sports            |  50    |   5      |  5   | 60
      Politics          |  3     |  45      |  2   | 50
      Tech              |  4     |   3      |  33  | 40
      ---------------------------------------------------
      Total             |  57    |  53      |  40  | 150
    

From this, we calculate metrics per category. For example, for Sports:

• Precision = TP / (TP + FP) = 50 / (50 + 5 + 4) = 50 / 59 ≈ 0.847
• Recall = TP / (TP + FN) = 50 / (50 + 5 + 5) = 50 / 60 ≈ 0.833

Precision vs Recall tradeoff with examples

If you want to avoid wrongly labeling documents (false positives), focus on high precision. For example, in legal document sorting, wrongly labeling a contract as a lawsuit is bad.

If you want to find all documents of a category (avoid false negatives), focus on high recall. For example, in spam detection, missing spam emails is worse than wrongly marking some good emails.

Balancing both with F1 score helps when both errors matter.

What "good" vs "bad" metric values look like

Good: Precision and recall above 0.85 means the model correctly finds and labels most documents with few mistakes.

Bad: Precision or recall below 0.5 means many documents are mislabeled or missed, making the model unreliable.

Accuracy alone can be misleading if categories are unbalanced.

Common pitfalls in metrics

• Accuracy paradox: High accuracy but poor recall on rare categories.
• Data leakage: When test data leaks into training, metrics look better but model fails in real use.
• Overfitting: Very high training metrics but low test metrics means model memorizes instead of learning.

Self-check question

Your text classification model has 98% accuracy but only 12% recall on the "urgent" category. Is it good for production?

Answer: No. The model misses 88% of urgent documents, which is risky. High accuracy is misleading because "urgent" documents are rare but important. You should improve recall before using it.

Key Result

Precision and recall are key to evaluate text classification because they show how well the model finds and labels each document category.

Practice

1. Why do we use text classification in organizing documents?

easy

A. To automatically group documents by their content

B. To delete documents that are not useful

C. To translate documents into different languages

D. To create new documents from existing ones

Solution

1. Step 1: Understand the purpose of text classification

   Text classification is used to sort or group documents based on what they talk about.

2. Step 2: Identify the correct use case

   Among the options, only grouping documents by content matches the purpose of text classification.

3. Final Answer:

   To automatically group documents by their content -> Option A

4. Quick Check:

   Text classification = grouping documents [OK]

Hint: Text classification groups by content, not deletes or translates [OK]

Common Mistakes:

• Confusing classification with translation
• Thinking classification deletes documents
• Assuming classification creates new documents

2. Which of the following is the correct way to describe text classification?

easy

A. It removes stop words from text

B. It translates text into numbers for storage

C. It assigns labels to text based on content

D. It generates new text from existing text

Solution

1. Step 1: Define text classification

   Text classification means giving a label or category to a piece of text based on what it contains.

2. Step 2: Match the definition to options

   Only assigning labels based on content matches the definition of text classification.

3. Final Answer:

   It assigns labels to text based on content -> Option C

4. Quick Check:

   Assign labels = classification [OK]

Hint: Classification means labeling, not translating or generating [OK]

Common Mistakes:

• Mixing classification with text preprocessing
• Confusing classification with text generation
• Thinking classification is about data storage

3. Given this Python code snippet for text classification, what will be the output?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ['I love cats', 'I hate rain', 'Cats are great', 'Rain is bad']
labels = ['positive', 'negative', 'positive', 'negative']

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(X, labels)

new_text = ['I love rain']
X_new = vectorizer.transform(new_text)
prediction = model.predict(X_new)
print(prediction[0])

medium

A. negative

B. positive

C. neutral

D. error

Solution

1. Step 1: Understand training data and labels

   The model learns 'I love cats' and 'Cats are great' as positive, 'I hate rain' and 'Rain is bad' as negative.

2. Step 2: Predict label for 'I love rain'

   The word 'love' appears in positive examples, and 'rain' appears in negative examples. The model weighs 'love' more strongly positive, so prediction is 'positive'.

3. Final Answer:

   positive -> Option B

4. Quick Check:

   Model predicts 'positive' for 'I love rain' [OK]

Hint: Words linked to positive examples influence prediction [OK]

Common Mistakes:

• Assuming 'love' always makes prediction positive
• Ignoring word frequency impact
• Expecting neutral label which is not in training

4. Find the error in this text classification code snippet:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ['happy day', 'sad night']
labels = ['positive', 'negative']

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(texts, labels)  # Error here

new_text = ['happy night']
X_new = vectorizer.transform(new_text)
prediction = model.predict(X_new)
print(prediction[0])

medium

A. Transforming new_text before vectorizing

B. Missing import for MultinomialNB

C. Labels list is empty

D. Using texts instead of X in model.fit

Solution

1. Step 1: Check model.fit inputs

   Model expects numeric features (X), but texts (strings) are passed instead.

2. Step 2: Correct the input to model.fit

   Replace texts with X (vectorized data) to fix the error.

3. Final Answer:

   Using texts instead of X in model.fit -> Option D

4. Quick Check:

   model.fit needs numeric input X [OK]

Hint: Model.fit needs vectorized data, not raw text [OK]

Common Mistakes:

• Passing raw text instead of vectorized features
• Ignoring error messages about input types
• Confusing transform and fit_transform

5. You want to classify news articles into categories like 'sports', 'politics', and 'technology'. Which approach best explains why text classification helps here?

hard

A. It learns patterns from labeled articles to predict categories for new articles

B. It translates articles into multiple languages for wider reach

C. It summarizes articles to reduce reading time

D. It deletes irrelevant articles automatically

Solution

1. Step 1: Understand the goal of classifying news articles

   The goal is to assign correct categories to new articles based on past examples.

2. Step 2: Identify how text classification achieves this

   Text classification learns from labeled data patterns to predict categories for unseen articles.

3. Final Answer:

   It learns patterns from labeled articles to predict categories for new articles -> Option A

4. Quick Check:

   Learning from examples = classification [OK]

Hint: Classification learns from examples to label new data [OK]

Common Mistakes:

• Confusing classification with translation or summarization
• Thinking classification deletes data
• Assuming classification creates content