N-gram language models in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the purpose of N-gram models

What is the main purpose of using an N-gram language model in natural language processing?

A To generate images from text descriptions using neural networks

B To translate text from one language to another using deep learning

C To cluster documents based on their topics using unsupervised learning

D To predict the next word in a sequence based on the previous N-1 words

💡 Hint

Think about what N-gram models estimate about word sequences.

N-gram models estimate the probability of a word given the previous N-1 words, helping predict the next word in a sequence.

❓ Predict Output

intermediate

Output of a bigram probability calculation

Given the sentence: 'I love machine learning', and the bigram counts: {('I', 'love'): 3, ('love', 'machine'): 2, ('machine', 'learning'): 4}, what is the bigram probability P('machine'|'love') using maximum likelihood estimation?

NLP

bigram_counts = {('I', 'love'): 3, ('love', 'machine'): 2, ('machine', 'learning'): 4}
unigram_counts = {'I': 3, 'love': 4, 'machine': 4}
prob = bigram_counts[('love', 'machine')] / unigram_counts['love']
print(prob)

A 0.5

B 0.33

C 0.67

D 1.0

💡 Hint

Divide the count of the bigram by the count of the first word in the bigram.

The bigram probability P('machine'|'love') = count('love machine') / count('love') = 2 / 4 = 0.5

❓ Hyperparameter

advanced

Choosing the value of N in N-gram models

Which of the following is a common trade-off when increasing the value of N in an N-gram language model?

A Higher N always improves model accuracy without any drawbacks

B Higher N leads to better context capture but requires more data and increases sparsity

C Higher N reduces model size and speeds up training

D Higher N eliminates the need for smoothing techniques

💡 Hint

Think about how longer sequences affect data requirements and sparsity.

Increasing N captures more context but causes data sparsity and needs more training data, making option B correct.

❓ Metrics

advanced

Evaluating N-gram language models with perplexity

If an N-gram language model has a perplexity of 50 on a test set, what does this indicate about the model's performance?

A The model is uncertain and predicts the test data poorly

B The model has zero error on the test data

C The model is very confident and predicts the test data well

D The model's predictions are random and unrelated to the test data

💡 Hint

Lower perplexity means better prediction; higher means worse.

Perplexity measures how well a model predicts data; higher perplexity means more uncertainty and worse performance.

🔧 Debug

expert

Identifying the error in smoothing implementation

Consider this code snippet for add-one smoothing in a bigram model: ```python bigram_counts = {('the', 'cat'): 3, ('cat', 'sat'): 2} unigram_counts = {'the': 5, 'cat': 4} vocab_size = 10 word1 = 'cat' word2 = 'sat' prob = (bigram_counts.get((word1, word2), 0) + 1) / (unigram_counts[word1] + vocab_size) print(prob) ``` What error will this code raise when calculating the probability for the bigram ('sat', 'on')?

A ZeroDivisionError because unigram_counts[word1] is zero

B TypeError because bigram_counts.get returns None

C KeyError because 'sat' is not in unigram_counts

D No error, code runs correctly

💡 Hint

Check if all words used as keys exist in unigram_counts.

If 'sat' is not in unigram_counts, accessing unigram_counts[word1] raises KeyError.

Practice

1. What does an n-gram language model primarily do?

easy

A. Predict the next word based on previous words

B. Translate text from one language to another

C. Generate images from text descriptions

D. Detect the sentiment of a sentence

Solution

1. Step 1: Understand the purpose of n-gram models

   N-gram models look at sequences of words to predict what comes next.

2. Step 2: Identify the main function

   They use previous words to guess the next word in a sentence.

3. Final Answer:

   Predict the next word based on previous words -> Option A

4. Quick Check:

   N-gram models predict next word = A [OK]

Hint: N-grams predict next word from previous words [OK]

Common Mistakes:

• Confusing n-gram with translation models
• Thinking n-grams generate images
• Mixing up sentiment analysis with n-grams

2. Which of the following is the correct way to represent a bigram from the sentence 'I love AI'?

easy

A. ('AI', 'love')

B. ('I', 'love')

C. ('love', 'AI', 'I')

D. ('I', 'AI')

Solution

1. Step 1: Understand bigrams

   Bigrams are pairs of consecutive words in a sentence.

2. Step 2: Extract bigrams from 'I love AI'

   The pairs are ('I', 'love') and ('love', 'AI'). ('I', 'love') shows a correct bigram.

3. Final Answer:

   ('I', 'love') -> Option B

4. Quick Check:

   Bigram = consecutive word pairs = C [OK]

Hint: Bigrams are pairs of consecutive words [OK]

Common Mistakes:

• Including three words instead of two
• Mixing word order in pairs
• Selecting non-consecutive words

3. Given the sentence 'the cat sat on the mat', what is the count of the trigram ('the', 'cat', 'sat')?

medium

A. 0

B. 2

C. 1

D. 3

Solution

1. Step 1: Identify trigrams in the sentence

   Trigrams are sequences of three consecutive words. The trigrams are: ('the', 'cat', 'sat'), ('cat', 'sat', 'on'), ('sat', 'on', 'the'), ('on', 'the', 'mat').

2. Step 2: Count the trigram ('the', 'cat', 'sat')

   This trigram appears once at the start of the sentence.

3. Final Answer:

   1 -> Option C

4. Quick Check:

   Trigram count = 1 [OK]

Hint: Count exact three-word sequences in order [OK]

Common Mistakes:

• Counting non-consecutive words
• Confusing bigrams with trigrams
• Overcounting repeated words

4. Consider this Python code snippet to generate bigrams from a list of words:

words = ['hello', 'world', 'hello']
bigrams = [(words[i], words[i+1]) for i in range(len(words))]

What error will this code produce?

medium

A. No error, code runs correctly

B. SyntaxError: invalid syntax

C. TypeError: unsupported operand type(s)

D. IndexError: list index out of range

Solution

1. Step 1: Analyze the loop range

   The loop runs from 0 to len(words)-1, which is 0 to 2 for 3 words.

2. Step 2: Check index access inside loop

   At i=2, words[i+1] tries to access words[3], which is out of range, causing IndexError.

3. Final Answer:

   IndexError: list index out of range -> Option D

4. Quick Check:

   Loop index exceeds list length = D [OK]

Hint: Check loop range when accessing i+1 index [OK]

Common Mistakes:

• Using full length in range causing out-of-bounds
• Assuming no error without testing
• Confusing syntax errors with runtime errors

5. You want to build a trigram model from a text corpus but notice many rare trigrams cause sparse data issues. Which technique can help improve your model's predictions?

hard

A. Use smoothing methods like Laplace smoothing

B. Increase the n in n-gram to 5-grams

C. Remove all trigrams that appear less than 10 times

D. Ignore the problem and use raw counts

Solution

1. Step 1: Understand sparse data in n-gram models

   Rare trigrams cause zero or low counts, making predictions unreliable.

2. Step 2: Identify smoothing techniques

   Smoothing like Laplace adds small counts to all n-grams, reducing zero probabilities and improving predictions.

3. Final Answer:

   Use smoothing methods like Laplace smoothing -> Option A

4. Quick Check:

   Smoothing reduces sparse data issues = A [OK]

Hint: Apply smoothing to handle rare n-grams [OK]

Common Mistakes:

• Increasing n worsens sparsity
• Removing rare n-grams loses useful info
• Ignoring sparsity leads to poor predictions