Model selection for tasks in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Model selection for tasks

Which metric matters for model selection and WHY

Choosing the right metric depends on the task you want your model to do. For example, if you want to classify emails as spam or not, precision matters because you want to avoid marking good emails as spam. If you want to find all spam emails, recall is important. For tasks like language translation or text generation, metrics like BLEU or ROUGE measure how close the output is to human language. Always pick a metric that matches what you care about in your task.

Confusion matrix example for classification tasks

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |

    Example:
    TP = 70, FP = 10, TN = 900, FN = 20
    Total samples = 70 + 10 + 900 + 20 = 1000
    

From this, you can calculate precision = 70 / (70 + 10) = 0.875 and recall = 70 / (70 + 20) = 0.778.

Precision vs Recall tradeoff with examples

Imagine a model that detects cancer from scans. Missing a cancer case (low recall) is very bad because the patient won't get treatment. So, recall is more important here. But if the model marks many healthy people as cancer (low precision), it causes stress and extra tests.

In contrast, a spam filter should have high precision to avoid losing important emails, even if it misses some spam (lower recall).

Choosing the right balance depends on what mistakes cost more in your task.

What good vs bad metric values look like

For a spam filter:

• Good: Precision > 0.9, Recall around 0.7 (few false spam)
• Bad: Precision < 0.5 (many good emails marked spam)

For a cancer detector:

• Good: Recall > 0.9 (few missed cancers)
• Bad: Recall < 0.5 (many cancers missed)

Accuracy alone can be misleading if classes are imbalanced.

Common pitfalls in metrics

• Accuracy paradox: High accuracy can happen if one class dominates, but the model ignores the minority class.
• Data leakage: When test data leaks into training, metrics look too good but model fails in real life.
• Overfitting: Model performs great on training but poorly on new data, metrics differ greatly between train and test.
• Wrong metric choice: Using accuracy for imbalanced data or BLEU for tasks it doesn't fit.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous because fraud goes undetected. High accuracy is misleading here because fraud is rare, so the model mostly predicts non-fraud correctly but fails at the important task.

Key Result

Choosing the right metric depends on the task; precision and recall tradeoffs are key for model selection.

Practice

1. Which type of model is best suited for a task where you want to classify emails as spam or not spam?

easy

A. A regression model

B. A clustering model

C. A classification model

D. A dimensionality reduction model

Solution

1. Step 1: Understand the task type

   Classifying emails as spam or not spam is a task where the output is a category or label.

2. Step 2: Match task to model type

   Classification models are designed to assign categories to inputs, so they fit this task best.

3. Final Answer:

   A classification model -> Option C

4. Quick Check:

   Classification = Spam detection [OK]

Hint: Classification = sorting into groups or labels [OK]

Common Mistakes:

• Confusing classification with regression
• Choosing clustering which is unsupervised
• Selecting dimensionality reduction which is for feature simplification

2. Which of the following is the correct way to import a pre-trained BERT model using the Hugging Face Transformers library in Python?

easy

A. from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased')

B. import bert; model = bert.load('bert-base-uncased')

C. from transformers import BERT; model = BERT.load_pretrained('bert-base-uncased')

D. import transformers; model = transformers.Bert.load('bert-base-uncased')

Solution

1. Step 1: Recall correct import syntax

   The Hugging Face library uses 'from transformers import BertModel' to import the BERT model class.

2. Step 2: Use the correct method to load pre-trained weights

   The method to load pre-trained weights is 'from_pretrained' called on the model class.

3. Final Answer:

   from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased') -> Option A

4. Quick Check:

   Correct import and method = from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased') [OK]

Hint: Use 'from transformers import ModelName' and 'from_pretrained' method [OK]

Common Mistakes:

• Using incorrect class names like BERT instead of BertModel
• Calling non-existent methods like load_pretrained
• Wrong import statements or missing 'from'

3. Consider the following Python code using scikit-learn for sentiment analysis. What will be the printed accuracy after training and testing the model?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

texts = ['I love this movie', 'This movie is bad', 'Amazing film', 'Terrible acting', 'Great plot', 'Worst movie ever']
labels = [1, 0, 1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.33, random_state=42)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

model = LogisticRegression()
model.fit(X_train_vec, y_train)

accuracy = model.score(X_test_vec, y_test)
print(round(accuracy, 2))

medium

A. 0.67

B. 1.0

C. 0.33

D. 0.5

Solution

1. Step 1: Understand data split and labels

   With random_state=42 and test_size=0.33, the test set contains 2 samples: 'Amazing film' (label 1) and 'Worst movie ever' (label 0).

2. Step 2: Predict and check accuracy

   Logistic Regression on simple text features predicts one sample correctly and one incorrectly, resulting in accuracy 0.5.

3. Final Answer:

   0.5 -> Option D

4. Quick Check:

   Accuracy = 1 correct / 2 total = 0.5 [OK]

Hint: Accuracy = correct predictions / total test samples [OK]

Common Mistakes:

• Assuming perfect accuracy without checking data split
• Confusing training accuracy with test accuracy
• Ignoring random_state effect on split

4. You tried to train a text classification model but got an error: ValueError: Found input variables with inconsistent numbers of samples. What is the most likely cause?

medium

A. The learning rate is too high

B. The input texts and labels have different lengths

C. The model architecture is incorrect

D. The tokenizer is missing

Solution

1. Step 1: Understand the error message

   The error says input variables have inconsistent sample counts, meaning the number of texts and labels differ.

2. Step 2: Identify cause in data preparation

   If texts and labels lists have different lengths, model training fails because each input needs a matching label.

3. Final Answer:

   The input texts and labels have different lengths -> Option B

4. Quick Check:

   Mismatch in input and label counts causes this error [OK]

Hint: Check if inputs and labels have same number of samples [OK]

Common Mistakes:

• Blaming model architecture for data size errors
• Ignoring data preprocessing steps
• Assuming learning rate or tokenizer causes this error

5. You want to build a model to generate text summaries from long articles. Which model type should you select for this task?

hard

A. Sequence-to-sequence model with attention

B. Binary classification model

C. Clustering model

D. Regression model

Solution

1. Step 1: Understand the task requirements

   Text summarization requires converting a long sequence (article) into a shorter sequence (summary).

2. Step 2: Identify suitable model architecture

   Sequence-to-sequence models with attention are designed to handle input and output sequences of different lengths effectively.

3. Final Answer:

   Sequence-to-sequence model with attention -> Option A

4. Quick Check:

   Summarization = sequence-to-sequence with attention [OK]

Hint: Summarization needs sequence-to-sequence models [OK]

Common Mistakes:

• Choosing classification or regression which don't generate sequences
• Selecting clustering which is unsupervised grouping
• Ignoring the need for attention in long text tasks