Part-of-speech tagging in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Part-of-speech tagging

Which metric matters for Part-of-speech tagging and WHY

For part-of-speech (POS) tagging, accuracy is the main metric. This is because POS tagging is a classification task where each word is assigned one correct tag. Accuracy tells us the percentage of words tagged correctly out of all words. Since every word must have exactly one tag, accuracy directly shows how well the model is doing overall.

Sometimes, per-tag precision and recall are also useful to understand how well the model predicts specific tags, especially if some tags are rare or more important.

Confusion matrix example for POS tagging

Imagine a simple POS tagger that predicts three tags: Noun (N), Verb (V), and Adjective (Adj). Here is a confusion matrix for 100 words:

      | Predicted N | Predicted V | Predicted Adj |
    ---------------------------------------------
    N |     40      |      5      |       5       |
    V |      3      |     30      |       2       |
    Adj|     2      |      3      |      10       |
    ---------------------------------------------
    Total words = 100
    

From this matrix:

• True Positives for Noun = 40
• False Positives for Noun = 3 + 2 = 5 (words wrongly predicted as Noun)
• False Negatives for Noun = 5 + 5 = 10 (Nouns predicted as other tags)

Accuracy = (40 + 30 + 10) / 100 = 80%

Precision vs Recall tradeoff with examples

In POS tagging, precision and recall per tag help understand errors:

• Precision for a tag means: Of all words predicted as that tag, how many were correct?
• Recall for a tag means: Of all words that truly have that tag, how many did the model find?

Example: For the Verb tag, if precision is high but recall is low, the model is very sure when it says a word is a verb but misses many verbs. This might happen if the model is cautious and only tags clear verbs.

For POS tagging, a balance is important because missing tags (low recall) or wrongly tagging words (low precision) both reduce usefulness.

What "good" vs "bad" metric values look like for POS tagging

Good metrics:

• Accuracy above 90% on a balanced dataset means most words are tagged correctly.
• Precision and recall above 85% for common tags like Noun and Verb.
• Consistent performance across tags, not just on frequent ones.

Bad metrics:

• Accuracy below 70% means many words are tagged wrong.
• Very low recall for some tags means the model misses many words of that type.
• High precision but very low recall or vice versa indicates imbalance and poor tagging quality.

Common pitfalls in POS tagging metrics

• Ignoring rare tags: Some tags appear rarely but are important. Ignoring their performance hides problems.
• Accuracy paradox: If the dataset has many nouns, a model tagging everything as noun can get high accuracy but is useless.
• Data leakage: Using test sentences seen during training inflates accuracy falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data, not generalizing.

Self-check question

Your POS tagging model has 98% accuracy but only 12% recall on the "Verb" tag. Is this good for production? Why or why not?

Answer: No, it is not good. The model misses most verbs (low recall), which means many verbs are tagged incorrectly or missed. Even though overall accuracy is high, the poor recall on verbs can cause serious problems in understanding sentences. The model needs improvement to better detect verbs.

Key Result

Accuracy is key for POS tagging, but per-tag precision and recall reveal detailed strengths and weaknesses.

Practice

1. What is the main purpose of part-of-speech tagging in natural language processing?

easy

A. To label each word with its grammatical role in a sentence

B. To translate text from one language to another

C. To count the number of words in a sentence

D. To generate new sentences automatically

Solution

1. Step 1: Understand the role of part-of-speech tagging

   Part-of-speech tagging assigns labels like noun, verb, adjective to each word, showing its grammatical role.

2. Step 2: Compare with other options

   Translation, word counting, and sentence generation are different NLP tasks unrelated to POS tagging.

3. Final Answer:

   To label each word with its grammatical role in a sentence -> Option A

4. Quick Check:

   POS tagging = labeling word roles [OK]

Hint: POS tagging means labeling words by their grammar role [OK]

Common Mistakes:

• Confusing POS tagging with translation
• Thinking POS tagging counts words
• Assuming POS tagging generates sentences

2. Which Python code correctly uses NLTK to perform part-of-speech tagging on the sentence 'I love AI'?

easy

A. import nltk
nltk.pos_tag(['I', 'love', 'AI'])

B. import nltk
nltk.tag_pos(['I', 'love', 'AI'])

C. import nltk
nltk.pos_tag('I love AI')

D. import nltk
nltk.pos_tag(['I love AI'])

Solution

1. Step 1: Check correct function and input type

   The correct function is pos_tag and it expects a list of words, not a string.

2. Step 2: Analyze each option

   import nltk nltk.pos_tag(['I', 'love', 'AI']) uses pos_tag with a list of words, which is correct. import nltk nltk.tag_pos(['I', 'love', 'AI']) uses a wrong function name. import nltk nltk.pos_tag('I love AI') passes a string instead of a list. import nltk nltk.pos_tag(['I love AI']) passes a list with one string, not separate words.

3. Final Answer:

   import nltk nltk.pos_tag(['I', 'love', 'AI']) -> Option A

4. Quick Check:

   pos_tag + list of words = correct syntax [OK]

Hint: Use pos_tag with a list of words, not a single string [OK]

Common Mistakes:

• Passing a string instead of a list
• Using incorrect function name
• Passing a list with one combined string

3. What is the output of the following Python code using NLTK's pos_tag?

import nltk
sentence = ['She', 'runs', 'fast']
tagged = nltk.pos_tag(sentence)
print(tagged)

medium

A. [('She', 'DT'), ('runs', 'VB'), ('fast', 'RB')]

B. [('She', 'NN'), ('runs', 'NN'), ('fast', 'JJ')]

C. [('She', 'PRP'), ('runs', 'VBZ'), ('fast', 'RB')]

D. [('She', 'PRP'), ('runs', 'VBD'), ('fast', 'RB')]

Solution

1. Step 1: Understand POS tags for each word

   'She' is a pronoun (PRP), 'runs' is a verb in present tense third person singular (VBZ), 'fast' is an adverb (RB).

2. Step 2: Match tags with options

   [('She', 'PRP'), ('runs', 'VBZ'), ('fast', 'RB')] matches these tags exactly. Other options have incorrect tags like noun (NN), determiner (DT), or past tense verb (VBD).

3. Final Answer:

   [('She', 'PRP'), ('runs', 'VBZ'), ('fast', 'RB')] -> Option C

4. Quick Check:

   Pronoun + present verb + adverb = [('She', 'PRP'), ('runs', 'VBZ'), ('fast', 'RB')] [OK]

Hint: Know common POS tags: PRP=pronoun, VBZ=verb present, RB=adverb [OK]

Common Mistakes:

• Confusing verb tenses VBZ vs VBD
• Mixing pronouns with nouns
• Mislabeling adverbs as adjectives

4. The following code throws an error. What is the most likely cause?

import nltk
sentence = 'He is happy'
tagged = nltk.pos_tag(sentence)
print(tagged)

medium

A. The nltk module is not imported correctly

B. The sentence variable should be a tuple, not a string

C. pos_tag requires a second argument specifying the language

D. The input to pos_tag should be a list of words, not a string

Solution

1. Step 1: Check input type for pos_tag

   pos_tag expects a list of words, but here a single string is passed, which causes an error.

2. Step 2: Verify other options

   nltk is imported correctly, pos_tag does not require a language argument, and input as tuple is not required.

3. Final Answer:

   The input to pos_tag should be a list of words, not a string -> Option D

4. Quick Check:

   pos_tag input must be list, not string [OK]

Hint: Always pass a list of words to pos_tag, not a string [OK]

Common Mistakes:

• Passing a string instead of a list
• Assuming pos_tag needs language argument
• Confusing input types (tuple vs list)

5. You want to tag parts of speech in a sentence but also handle unknown words gracefully. Which approach best improves POS tagging accuracy for new words?

hard

A. Manually assign tags to each unknown word before tagging

B. Use a POS tagger with a built-in model trained on large diverse text

C. Ignore unknown words during tagging to avoid errors

D. Replace unknown words with a fixed placeholder before tagging

Solution

1. Step 1: Understand handling unknown words in POS tagging

   Taggers trained on large, diverse datasets can predict tags for new words based on context and patterns.

2. Step 2: Evaluate other options

   Manually tagging unknown words is impractical, ignoring them loses information, and replacing with placeholders removes context.

3. Final Answer:

   Use a POS tagger with a built-in model trained on large diverse text -> Option B

4. Quick Check:

   Robust model with training data handles unknown words best [OK]

Hint: Choose taggers trained on big data to handle new words well [OK]

Common Mistakes:

• Trying to manually tag unknown words
• Ignoring unknown words instead of tagging
• Replacing words loses sentence meaning