N-gram language models in NLP - Model Metrics & Evaluation

Metrics & Evaluation - N-gram language models

Which metric matters for N-gram language models and WHY

N-gram language models predict the next word based on previous words. The key metric is Perplexity. It measures how well the model predicts a sample. Lower perplexity means the model is better at guessing the next word, like guessing the next word in a sentence with less surprise.

Perplexity is important because it directly shows how uncertain the model is. A model with low perplexity is more confident and accurate in its predictions.

Confusion matrix or equivalent visualization

N-gram models predict probabilities for many possible next words, so a confusion matrix is not typical. Instead, we look at Perplexity, which is calculated from the probabilities the model assigns to the correct next words.

Perplexity = 2^(- (1/N) * sum(log2 P(w_i | context)))

Where:
- N is the number of words
- P(w_i | context) is the predicted probability of the actual next word

Example:
If the model predicts the next word with probabilities: 
"cat"=0.5, "dog"=0.3, "bird"=0.2,
and the actual next word is "cat", the log probability is log2(0.5) = -1.
Perplexity measures the average of these log probabilities over the test set.
    

Precision vs Recall tradeoff with concrete examples

Precision and recall are not commonly used for N-gram language models because predictions are probabilistic over many words, not binary decisions.

Instead, the tradeoff is between model complexity and data sparsity. Using larger N-grams (like 4-grams) can improve prediction but needs more data. Smaller N-grams (like bigrams) are simpler but less precise.

Example: A trigram model might predict "I love cats" better than a bigram model, but if you don't have enough data, the trigram model might guess poorly because it never saw "I love" before.

What "good" vs "bad" metric values look like for N-gram models

Good: Low perplexity, close to 10 or less on a typical English dataset means the model predicts well.

Bad: High perplexity, like 100 or more, means the model is very uncertain and guesses poorly.

Note: Perplexity depends on dataset size and vocabulary. Comparing perplexity only makes sense between models on the same data.

Common pitfalls in metrics for N-gram language models

• Data sparsity: Many N-grams never appear in training, causing zero probabilities and infinite perplexity if not smoothed.
• Overfitting: Very large N-grams memorize training data but fail on new text, causing low training perplexity but high test perplexity.
• Ignoring smoothing: Without smoothing techniques, the model assigns zero probability to unseen N-grams, breaking perplexity calculation.
• Comparing perplexity across datasets: Perplexity values are not comparable if datasets differ in size or vocabulary.

Self-check question

Your trigram model has a perplexity of 150 on the test set but 20 on the training set. Is this model good? Why or why not?

Answer: No, this model is not good. The low training perplexity means it learned the training data well, but the very high test perplexity means it does not predict new text well. This suggests overfitting and poor generalization.

Key Result

Perplexity is the key metric for N-gram models; lower perplexity means better next-word prediction.

Practice

1. What does an n-gram language model primarily do?

easy

A. Predict the next word based on previous words

B. Translate text from one language to another

C. Generate images from text descriptions

D. Detect the sentiment of a sentence

Solution

1. Step 1: Understand the purpose of n-gram models

   N-gram models look at sequences of words to predict what comes next.

2. Step 2: Identify the main function

   They use previous words to guess the next word in a sentence.

3. Final Answer:

   Predict the next word based on previous words -> Option A

4. Quick Check:

   N-gram models predict next word = A [OK]

Hint: N-grams predict next word from previous words [OK]

Common Mistakes:

• Confusing n-gram with translation models
• Thinking n-grams generate images
• Mixing up sentiment analysis with n-grams

2. Which of the following is the correct way to represent a bigram from the sentence 'I love AI'?

easy

A. ('AI', 'love')

B. ('I', 'love')

C. ('love', 'AI', 'I')

D. ('I', 'AI')

Solution

1. Step 1: Understand bigrams

   Bigrams are pairs of consecutive words in a sentence.

2. Step 2: Extract bigrams from 'I love AI'

   The pairs are ('I', 'love') and ('love', 'AI'). ('I', 'love') shows a correct bigram.

3. Final Answer:

   ('I', 'love') -> Option B

4. Quick Check:

   Bigram = consecutive word pairs = C [OK]

Hint: Bigrams are pairs of consecutive words [OK]

Common Mistakes:

• Including three words instead of two
• Mixing word order in pairs
• Selecting non-consecutive words

3. Given the sentence 'the cat sat on the mat', what is the count of the trigram ('the', 'cat', 'sat')?

medium

A. 0

B. 2

C. 1

D. 3

Solution

1. Step 1: Identify trigrams in the sentence

   Trigrams are sequences of three consecutive words. The trigrams are: ('the', 'cat', 'sat'), ('cat', 'sat', 'on'), ('sat', 'on', 'the'), ('on', 'the', 'mat').

2. Step 2: Count the trigram ('the', 'cat', 'sat')

   This trigram appears once at the start of the sentence.

3. Final Answer:

   1 -> Option C

4. Quick Check:

   Trigram count = 1 [OK]

Hint: Count exact three-word sequences in order [OK]

Common Mistakes:

• Counting non-consecutive words
• Confusing bigrams with trigrams
• Overcounting repeated words

4. Consider this Python code snippet to generate bigrams from a list of words:

words = ['hello', 'world', 'hello']
bigrams = [(words[i], words[i+1]) for i in range(len(words))]

What error will this code produce?

medium

A. No error, code runs correctly

B. SyntaxError: invalid syntax

C. TypeError: unsupported operand type(s)

D. IndexError: list index out of range

Solution

1. Step 1: Analyze the loop range

   The loop runs from 0 to len(words)-1, which is 0 to 2 for 3 words.

2. Step 2: Check index access inside loop

   At i=2, words[i+1] tries to access words[3], which is out of range, causing IndexError.

3. Final Answer:

   IndexError: list index out of range -> Option D

4. Quick Check:

   Loop index exceeds list length = D [OK]

Hint: Check loop range when accessing i+1 index [OK]

Common Mistakes:

• Using full length in range causing out-of-bounds
• Assuming no error without testing
• Confusing syntax errors with runtime errors

5. You want to build a trigram model from a text corpus but notice many rare trigrams cause sparse data issues. Which technique can help improve your model's predictions?

hard

A. Use smoothing methods like Laplace smoothing

B. Increase the n in n-gram to 5-grams

C. Remove all trigrams that appear less than 10 times

D. Ignore the problem and use raw counts

Solution

1. Step 1: Understand sparse data in n-gram models

   Rare trigrams cause zero or low counts, making predictions unreliable.

2. Step 2: Identify smoothing techniques

   Smoothing like Laplace adds small counts to all n-grams, reducing zero probabilities and improving predictions.

3. Final Answer:

   Use smoothing methods like Laplace smoothing -> Option A

4. Quick Check:

   Smoothing reduces sparse data issues = A [OK]

Hint: Apply smoothing to handle rare n-grams [OK]

Common Mistakes:

• Increasing n worsens sparsity
• Removing rare n-grams loses useful info
• Ignoring sparsity leads to poor predictions