Model selection for tasks in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is model selection in machine learning?

Model selection is the process of choosing the best machine learning model for a specific task based on performance and suitability.

beginner

Why is it important to select the right model for a task?

Choosing the right model ensures better accuracy, faster training, and efficient use of resources for the task at hand.

intermediate

Name two common criteria used to select models for NLP tasks.

Common criteria include accuracy (or other performance metrics) and computational efficiency (speed and resource use).

intermediate

How does the size of the dataset influence model selection?

Smaller datasets often require simpler models to avoid overfitting, while larger datasets can support more complex models.

intermediate

What role does the task type (e.g., classification, generation) play in model selection?

The task type determines which model architectures are suitable, such as classification models for labeling tasks and sequence-to-sequence models for generation.

Which factor is NOT typically considered in model selection?

A Model accuracy

B Model color

C Training time

D Dataset size

Model color is irrelevant; accuracy, training time, and dataset size are important factors.

For a small dataset, which model is usually better?

A A very deep neural network

B A model trained on unrelated data

C A model with many parameters

D A simple model

Simple models reduce overfitting risk on small datasets.

Which model type is best for text generation tasks?

A Sequence-to-sequence models

B Linear regression

C Decision trees

D Clustering algorithms

Sequence-to-sequence models handle input-output sequences well, ideal for generation.

What metric is commonly used to evaluate classification models?

A BLEU score

B Mean squared error

C Accuracy

D Perplexity

Accuracy measures correct predictions in classification tasks.

Why might computational efficiency matter in model selection?

A To reduce training and prediction time

B To make the model colorful

C To increase model complexity unnecessarily

D To ignore dataset size

Efficient models save time and resources during training and use.

Explain how dataset size affects your choice of model for an NLP task.

Think about how much data you have and how complex your model should be.

Describe why task type is important when selecting a model in NLP.

Consider what the model needs to do with the text.

Practice

1. Which type of model is best suited for a task where you want to classify emails as spam or not spam?

easy

A. A regression model

B. A clustering model

C. A classification model

D. A dimensionality reduction model

Solution

1. Step 1: Understand the task type

   Classifying emails as spam or not spam is a task where the output is a category or label.

2. Step 2: Match task to model type

   Classification models are designed to assign categories to inputs, so they fit this task best.

3. Final Answer:

   A classification model -> Option C

4. Quick Check:

   Classification = Spam detection [OK]

Hint: Classification = sorting into groups or labels [OK]

Common Mistakes:

• Confusing classification with regression
• Choosing clustering which is unsupervised
• Selecting dimensionality reduction which is for feature simplification

2. Which of the following is the correct way to import a pre-trained BERT model using the Hugging Face Transformers library in Python?

easy

A. from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased')

B. import bert; model = bert.load('bert-base-uncased')

C. from transformers import BERT; model = BERT.load_pretrained('bert-base-uncased')

D. import transformers; model = transformers.Bert.load('bert-base-uncased')

Solution

1. Step 1: Recall correct import syntax

   The Hugging Face library uses 'from transformers import BertModel' to import the BERT model class.

2. Step 2: Use the correct method to load pre-trained weights

   The method to load pre-trained weights is 'from_pretrained' called on the model class.

3. Final Answer:

   from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased') -> Option A

4. Quick Check:

   Correct import and method = from transformers import BertModel; model = BertModel.from_pretrained('bert-base-uncased') [OK]

Hint: Use 'from transformers import ModelName' and 'from_pretrained' method [OK]

Common Mistakes:

• Using incorrect class names like BERT instead of BertModel
• Calling non-existent methods like load_pretrained
• Wrong import statements or missing 'from'

3. Consider the following Python code using scikit-learn for sentiment analysis. What will be the printed accuracy after training and testing the model?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

texts = ['I love this movie', 'This movie is bad', 'Amazing film', 'Terrible acting', 'Great plot', 'Worst movie ever']
labels = [1, 0, 1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.33, random_state=42)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

model = LogisticRegression()
model.fit(X_train_vec, y_train)

accuracy = model.score(X_test_vec, y_test)
print(round(accuracy, 2))

medium

A. 0.67

B. 1.0

C. 0.33

D. 0.5

Solution

1. Step 1: Understand data split and labels

   With random_state=42 and test_size=0.33, the test set contains 2 samples: 'Amazing film' (label 1) and 'Worst movie ever' (label 0).

2. Step 2: Predict and check accuracy

   Logistic Regression on simple text features predicts one sample correctly and one incorrectly, resulting in accuracy 0.5.

3. Final Answer:

   0.5 -> Option D

4. Quick Check:

   Accuracy = 1 correct / 2 total = 0.5 [OK]

Hint: Accuracy = correct predictions / total test samples [OK]

Common Mistakes:

• Assuming perfect accuracy without checking data split
• Confusing training accuracy with test accuracy
• Ignoring random_state effect on split

4. You tried to train a text classification model but got an error: ValueError: Found input variables with inconsistent numbers of samples. What is the most likely cause?

medium

A. The learning rate is too high

B. The input texts and labels have different lengths

C. The model architecture is incorrect

D. The tokenizer is missing

Solution

1. Step 1: Understand the error message

   The error says input variables have inconsistent sample counts, meaning the number of texts and labels differ.

2. Step 2: Identify cause in data preparation

   If texts and labels lists have different lengths, model training fails because each input needs a matching label.

3. Final Answer:

   The input texts and labels have different lengths -> Option B

4. Quick Check:

   Mismatch in input and label counts causes this error [OK]

Hint: Check if inputs and labels have same number of samples [OK]

Common Mistakes:

• Blaming model architecture for data size errors
• Ignoring data preprocessing steps
• Assuming learning rate or tokenizer causes this error

5. You want to build a model to generate text summaries from long articles. Which model type should you select for this task?

hard

A. Sequence-to-sequence model with attention

B. Binary classification model

C. Clustering model

D. Regression model

Solution

1. Step 1: Understand the task requirements

   Text summarization requires converting a long sequence (article) into a shorter sequence (summary).

2. Step 2: Identify suitable model architecture

   Sequence-to-sequence models with attention are designed to handle input and output sequences of different lengths effectively.

3. Final Answer:

   Sequence-to-sequence model with attention -> Option A

4. Quick Check:

   Summarization = sequence-to-sequence with attention [OK]

Hint: Summarization needs sequence-to-sequence models [OK]

Common Mistakes:

• Choosing classification or regression which don't generate sequences
• Selecting clustering which is unsupervised grouping
• Ignoring the need for attention in long text tasks