Language modeling concept in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Language modeling concept

Which metric matters for language modeling and WHY

For language models, the main metric is Perplexity. It measures how well the model predicts the next word. A lower perplexity means the model is better at guessing the next word in a sentence, just like how a good friend can finish your sentence correctly. Perplexity is important because it directly shows how confident and accurate the model is in understanding language patterns.

Confusion matrix or equivalent visualization

Language modeling usually predicts many possible next words, so a confusion matrix is not typical. Instead, we look at Perplexity, which is calculated from the probabilities the model assigns to the correct next words.

Perplexity = 2^(- (1/N) * sum(log2 P(w_i | context)))

Where:
- N is the number of words
- P(w_i | context) is the predicted probability of the actual next word

Example:
If the model predicts the next word with 0.5 probability, perplexity contribution is 2^(-log2(0.5)) = 2^1 = 2
Lower perplexity means better prediction.
    

Precision vs Recall tradeoff with concrete examples

Precision and recall are less common for language modeling because it predicts probabilities over many words. But if we think about next word prediction as a classification task, there is a tradeoff:

• Precision: How often the predicted word is actually correct. High precision means the model rarely guesses wrong words.
• Recall: How many of the correct next words the model can predict. High recall means the model covers many possible correct words.

For example, in autocomplete on your phone, high precision avoids annoying wrong suggestions, while high recall helps suggest many useful words. Language models balance these by assigning probabilities to many words.

What "good" vs "bad" metric values look like for language modeling

Good: Perplexity close to 10 or lower on a test set means the model predicts next words well. It shows the model understands language patterns clearly.

Bad: Perplexity above 100 means the model is very confused and guesses poorly. It might be just picking words randomly or not learning from data.

Remember, perplexity depends on dataset size and complexity, so compare models on the same data.

Metrics pitfalls in language modeling

• Overfitting: Very low perplexity on training data but high on test data means the model memorizes instead of learning language rules.
• Data leakage: If test sentences appear in training, perplexity looks artificially low, hiding true performance.
• Ignoring context length: Short context can make perplexity look better but model may fail on longer sentences.
• Comparing across datasets: Perplexity values vary by dataset size and vocabulary, so only compare models on the same data.

Self-check question

Your language model has a perplexity of 50 on training data but 200 on test data. Is it good? Why or why not?

Answer: This is not good. The model performs well on training but poorly on test data, showing it memorized training sentences and cannot generalize to new text. It needs better training or regularization.

Key Result

Perplexity is the key metric for language modeling; lower values mean better next-word prediction.

Practice

1. What is the main goal of a language model in natural language processing?

easy

A. To predict the next word in a sentence

B. To translate text from one language to another

C. To count the number of words in a document

D. To summarize long paragraphs into short sentences

Solution

1. Step 1: Understand the purpose of language models

   Language models are designed to understand and predict text sequences.

2. Step 2: Identify the main task of language models

   The core task is to predict the next word based on previous words in a sentence.

3. Final Answer:

   To predict the next word in a sentence -> Option A

4. Quick Check:

   Language model goal = predict next word [OK]

Hint: Language models guess the next word in text [OK]

Common Mistakes:

• Confusing language modeling with translation
• Thinking language models only count words
• Assuming summarization is the main task

2. Which of the following is the correct way to represent a bigram language model probability for a sentence "I love AI"?

easy

A. P(I) * P(love) * P(AI)

B. P(I | AI) * P(love | I) * P(AI | love)

C. P(I | love) * P(love | AI) * P(AI)

D. P(I) * P(love | I) * P(AI | love)

Solution

1. Step 1: Recall bigram model definition

   A bigram model predicts each word based on the previous word, so probabilities are conditional.

2. Step 2: Apply bigram probabilities to the sentence

   The sentence probability is P(I) * P(love | I) * P(AI | love), starting with the first word's probability.

3. Final Answer:

   P(I) * P(love | I) * P(AI | love) -> Option D

4. Quick Check:

   Bigram = word depends on previous word [OK]

Hint: Bigram means each word depends on the one before [OK]

Common Mistakes:

• Multiplying independent word probabilities (unigram)
• Using wrong conditional order
• Confusing bigram with trigram or other models

3. Given the following unigram probabilities: P(I)=0.2, P(love)=0.1, P(AI)=0.05, what is the probability of the sentence "I love AI" under a unigram model?

medium

A. 0.01

B. 0.001

C. 0.35

D. 0.0001

Solution

1. Step 1: Understand unigram model calculation

   Unigram model assumes words are independent, so multiply their probabilities.

2. Step 2: Calculate sentence probability

   Multiply P(I) * P(love) * P(AI) = 0.2 * 0.1 * 0.05 = 0.001

3. Final Answer:

   0.001 -> Option B

4. Quick Check:

   Unigram multiply all word probs = 0.001 [OK]

Hint: Multiply all word probabilities for unigram [OK]

Common Mistakes:

• Adding probabilities instead of multiplying
• Using conditional probabilities (bigram) by mistake
• Incorrect multiplication order

4. Consider this Python code snippet for a bigram model probability calculation:

sentence = ['I', 'love', 'AI']
bigram_probs = {('I', 'love'): 0.3, ('love', 'AI'): 0.4}
prob = 1.0
for i in range(len(sentence)-1):
    prob *= bigram_probs[(sentence[i], sentence[i+1])]
print(prob)

What error will occur when running this code?

medium

A. No error, prints 0.12

B. TypeError due to wrong data type in multiplication

C. KeyError because the first word probability is missing

D. IndexError because of range length

Solution

1. Step 1: Analyze the loop and dictionary access

   The loop multiplies probabilities for bigrams in the sentence using bigram_probs dictionary keys.

2. Step 2: Check if all bigrams exist in dictionary

   bigram_probs lacks a probability for the first word alone, but code only uses pairs, so no missing keys for pairs.

3. Step 3: Re-examine the code logic

   All bigrams ('I','love') and ('love','AI') exist in dictionary, so no KeyError. No TypeError or IndexError expected.

4. Final Answer:

   No error, prints 0.12 -> Option A

5. Quick Check:

   All bigrams found, multiply 0.3*0.4=0.12 [OK]

Hint: Check if all keys exist before dictionary access [OK]

Common Mistakes:

• Assuming first word needs separate probability
• Confusing KeyError with IndexError
• Ignoring dictionary key structure

5. You want to build a trigram language model to predict the next word given two previous words. Which approach best handles the problem of unseen trigrams in your training data?

hard

A. Only use unigram probabilities for all predictions

B. Ignore unseen trigrams and assign zero probability

C. Use smoothing techniques like Kneser-Ney smoothing

D. Increase the training data size without smoothing

Solution

1. Step 1: Understand the unseen trigram problem

   Unseen trigrams cause zero probabilities, which harm model predictions.

2. Step 2: Identify solution to zero probability issue

   Smoothing techniques like Kneser-Ney adjust probabilities to handle unseen cases effectively.

3. Step 3: Evaluate other options

   Ignoring unseen trigrams or only using unigram probabilities lose context; increasing data alone may not solve sparsity.

4. Final Answer:

   Use smoothing techniques like Kneser-Ney smoothing -> Option C

5. Quick Check:

   Smoothing fixes zero probs for unseen trigrams [OK]

Hint: Use smoothing to avoid zero probabilities [OK]

Common Mistakes:

• Assigning zero probability to unseen trigrams
• Ignoring context by using only unigrams
• Relying solely on more data without smoothing