N-gram language models in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is an N-gram in language modeling?

An N-gram is a sequence of N words used together to predict the next word in a sentence. For example, a bigram is two words, a trigram is three words.

beginner

How does an N-gram language model predict the next word?

It looks at the previous N-1 words and calculates the probability of each possible next word based on how often those words appeared together in training data.

beginner

What is the difference between unigram, bigram, and trigram models?

Unigram models consider single words independently. Bigram models consider pairs of words. Trigram models consider triples of words to predict the next word.

intermediate

Why do N-gram models face the problem of data sparsity?

Because many word combinations may not appear in the training data, making it hard to estimate their probabilities accurately.

intermediate

What is smoothing in N-gram language models?

Smoothing is a technique to adjust probabilities so that unseen word sequences get a small, non-zero probability instead of zero, helping the model handle new phrases.

What does a trigram model use to predict the next word?

A The previous two words

B The previous three words

C Only the last word

D All words in the sentence

A trigram model uses the previous two words (N-1) to predict the next word.

Why is smoothing important in N-gram models?

A To remove rare words from the model

B To assign zero probability to unseen word sequences

C To increase the size of the training data

D To assign small probabilities to unseen word sequences

Smoothing assigns small non-zero probabilities to unseen sequences, preventing zero probability issues.

Which problem occurs because many word sequences are rare or missing in training data?

A Overfitting

B Data sparsity

C Underfitting

D Bias

Data sparsity happens when many word combinations are rare or missing, making probability estimation hard.

In a bigram model, what is the probability of a word based on?

A Random chance

B The word after it

C The word before it

D The entire sentence

A bigram model predicts a word based on the immediately preceding word.

Which of these is NOT a type of N-gram model?

A Quadrigram

B Unigram

C Trigram

D Bigram

The correct term is 'four-gram' or '4-gram', not 'quadrigram'.

Explain how an N-gram language model predicts the next word in a sentence.

Think about how you guess the next word when typing a message.

Describe the challenges N-gram models face and how smoothing helps.

Consider what happens when the model sees a new phrase it never learned.

Practice

1. What does an n-gram language model primarily do?

easy

A. Predict the next word based on previous words

B. Translate text from one language to another

C. Generate images from text descriptions

D. Detect the sentiment of a sentence

Solution

1. Step 1: Understand the purpose of n-gram models

   N-gram models look at sequences of words to predict what comes next.

2. Step 2: Identify the main function

   They use previous words to guess the next word in a sentence.

3. Final Answer:

   Predict the next word based on previous words -> Option A

4. Quick Check:

   N-gram models predict next word = A [OK]

Hint: N-grams predict next word from previous words [OK]

Common Mistakes:

• Confusing n-gram with translation models
• Thinking n-grams generate images
• Mixing up sentiment analysis with n-grams

2. Which of the following is the correct way to represent a bigram from the sentence 'I love AI'?

easy

A. ('AI', 'love')

B. ('I', 'love')

C. ('love', 'AI', 'I')

D. ('I', 'AI')

Solution

1. Step 1: Understand bigrams

   Bigrams are pairs of consecutive words in a sentence.

2. Step 2: Extract bigrams from 'I love AI'

   The pairs are ('I', 'love') and ('love', 'AI'). ('I', 'love') shows a correct bigram.

3. Final Answer:

   ('I', 'love') -> Option B

4. Quick Check:

   Bigram = consecutive word pairs = C [OK]

Hint: Bigrams are pairs of consecutive words [OK]

Common Mistakes:

• Including three words instead of two
• Mixing word order in pairs
• Selecting non-consecutive words

3. Given the sentence 'the cat sat on the mat', what is the count of the trigram ('the', 'cat', 'sat')?

medium

A. 0

B. 2

C. 1

D. 3

Solution

1. Step 1: Identify trigrams in the sentence

   Trigrams are sequences of three consecutive words. The trigrams are: ('the', 'cat', 'sat'), ('cat', 'sat', 'on'), ('sat', 'on', 'the'), ('on', 'the', 'mat').

2. Step 2: Count the trigram ('the', 'cat', 'sat')

   This trigram appears once at the start of the sentence.

3. Final Answer:

   1 -> Option C

4. Quick Check:

   Trigram count = 1 [OK]

Hint: Count exact three-word sequences in order [OK]

Common Mistakes:

• Counting non-consecutive words
• Confusing bigrams with trigrams
• Overcounting repeated words

4. Consider this Python code snippet to generate bigrams from a list of words:

words = ['hello', 'world', 'hello']
bigrams = [(words[i], words[i+1]) for i in range(len(words))]

What error will this code produce?

medium

A. No error, code runs correctly

B. SyntaxError: invalid syntax

C. TypeError: unsupported operand type(s)

D. IndexError: list index out of range

Solution

1. Step 1: Analyze the loop range

   The loop runs from 0 to len(words)-1, which is 0 to 2 for 3 words.

2. Step 2: Check index access inside loop

   At i=2, words[i+1] tries to access words[3], which is out of range, causing IndexError.

3. Final Answer:

   IndexError: list index out of range -> Option D

4. Quick Check:

   Loop index exceeds list length = D [OK]

Hint: Check loop range when accessing i+1 index [OK]

Common Mistakes:

• Using full length in range causing out-of-bounds
• Assuming no error without testing
• Confusing syntax errors with runtime errors

5. You want to build a trigram model from a text corpus but notice many rare trigrams cause sparse data issues. Which technique can help improve your model's predictions?

hard

A. Use smoothing methods like Laplace smoothing

B. Increase the n in n-gram to 5-grams

C. Remove all trigrams that appear less than 10 times

D. Ignore the problem and use raw counts

Solution

1. Step 1: Understand sparse data in n-gram models

   Rare trigrams cause zero or low counts, making predictions unreliable.

2. Step 2: Identify smoothing techniques

   Smoothing like Laplace adds small counts to all n-grams, reducing zero probabilities and improving predictions.

3. Final Answer:

   Use smoothing methods like Laplace smoothing -> Option A

4. Quick Check:

   Smoothing reduces sparse data issues = A [OK]

Hint: Apply smoothing to handle rare n-grams [OK]

Common Mistakes:

• Increasing n worsens sparsity
• Removing rare n-grams loses useful info
• Ignoring sparsity leads to poor predictions