Language modeling concept in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What does a language model predict?

In simple terms, what is the main task of a language model?

A Predict the next word in a sentence based on previous words

B Translate text from one language to another

C Identify the topic of a given text

D Summarize a long paragraph into a short sentence

💡 Hint

Think about what comes next when you read a sentence.

A language model learns to guess the next word by looking at the words before it. This helps computers understand and generate text.

❓ Predict Output

intermediate

Output of a simple bigram language model prediction

Given the following bigram probabilities, what is the predicted next word after 'I'?

{'I': {'am': 0.6, 'like': 0.3, 'have': 0.1}}

A 'is'

B 'like'

C 'have'

D 'am'

💡 Hint

Choose the word with the highest probability after 'I'.

The word 'am' has the highest probability (0.6) following 'I', so it is the predicted next word.

❓ Model Choice

advanced

Choosing a model type for language modeling

You want to build a language model that understands long sentences and context. Which model type is best?

A N-gram model with n=2

B Recurrent Neural Network (RNN)

C Bag-of-words model

D Simple linear regression

💡 Hint

Think about models that remember previous words in a sequence.

RNNs can remember information from earlier words, making them good for understanding long sentences and context.

❓ Hyperparameter

advanced

Effect of increasing vocabulary size in language models

What is a likely effect of increasing the vocabulary size in a language model?

A Model requires more memory and training data

B Model training becomes faster and simpler

C Model accuracy always decreases

D Model ignores rare words automatically

💡 Hint

Think about what happens when the model has to learn more words.

More vocabulary means more parameters and data needed to learn word patterns well, increasing memory and data requirements.

❓ Metrics

expert

Interpreting perplexity in language models

A language model has a perplexity score of 20 on a test set. What does this mean?

A The model is 20 times better than random guessing

B The model makes 20% errors in predictions

C On average, the model is as uncertain as choosing among 20 options

D The model predicts the next word with 20% probability

💡 Hint

Perplexity measures how surprised the model is by the test data.

Perplexity of 20 means the model is as uncertain as picking one word out of 20 equally likely choices on average.

Practice

1. What is the main goal of a language model in natural language processing?

easy

A. To predict the next word in a sentence

B. To translate text from one language to another

C. To count the number of words in a document

D. To summarize long paragraphs into short sentences

Solution

1. Step 1: Understand the purpose of language models

   Language models are designed to understand and predict text sequences.

2. Step 2: Identify the main task of language models

   The core task is to predict the next word based on previous words in a sentence.

3. Final Answer:

   To predict the next word in a sentence -> Option A

4. Quick Check:

   Language model goal = predict next word [OK]

Hint: Language models guess the next word in text [OK]

Common Mistakes:

• Confusing language modeling with translation
• Thinking language models only count words
• Assuming summarization is the main task

2. Which of the following is the correct way to represent a bigram language model probability for a sentence "I love AI"?

easy

A. P(I) * P(love) * P(AI)

B. P(I | AI) * P(love | I) * P(AI | love)

C. P(I | love) * P(love | AI) * P(AI)

D. P(I) * P(love | I) * P(AI | love)

Solution

1. Step 1: Recall bigram model definition

   A bigram model predicts each word based on the previous word, so probabilities are conditional.

2. Step 2: Apply bigram probabilities to the sentence

   The sentence probability is P(I) * P(love | I) * P(AI | love), starting with the first word's probability.

3. Final Answer:

   P(I) * P(love | I) * P(AI | love) -> Option D

4. Quick Check:

   Bigram = word depends on previous word [OK]

Hint: Bigram means each word depends on the one before [OK]

Common Mistakes:

• Multiplying independent word probabilities (unigram)
• Using wrong conditional order
• Confusing bigram with trigram or other models

3. Given the following unigram probabilities: P(I)=0.2, P(love)=0.1, P(AI)=0.05, what is the probability of the sentence "I love AI" under a unigram model?

medium

A. 0.01

B. 0.001

C. 0.35

D. 0.0001

Solution

1. Step 1: Understand unigram model calculation

   Unigram model assumes words are independent, so multiply their probabilities.

2. Step 2: Calculate sentence probability

   Multiply P(I) * P(love) * P(AI) = 0.2 * 0.1 * 0.05 = 0.001

3. Final Answer:

   0.001 -> Option B

4. Quick Check:

   Unigram multiply all word probs = 0.001 [OK]

Hint: Multiply all word probabilities for unigram [OK]

Common Mistakes:

• Adding probabilities instead of multiplying
• Using conditional probabilities (bigram) by mistake
• Incorrect multiplication order

4. Consider this Python code snippet for a bigram model probability calculation:

sentence = ['I', 'love', 'AI']
bigram_probs = {('I', 'love'): 0.3, ('love', 'AI'): 0.4}
prob = 1.0
for i in range(len(sentence)-1):
    prob *= bigram_probs[(sentence[i], sentence[i+1])]
print(prob)

What error will occur when running this code?

medium

A. No error, prints 0.12

B. TypeError due to wrong data type in multiplication

C. KeyError because the first word probability is missing

D. IndexError because of range length

Solution

1. Step 1: Analyze the loop and dictionary access

   The loop multiplies probabilities for bigrams in the sentence using bigram_probs dictionary keys.

2. Step 2: Check if all bigrams exist in dictionary

   bigram_probs lacks a probability for the first word alone, but code only uses pairs, so no missing keys for pairs.

3. Step 3: Re-examine the code logic

   All bigrams ('I','love') and ('love','AI') exist in dictionary, so no KeyError. No TypeError or IndexError expected.

4. Final Answer:

   No error, prints 0.12 -> Option A

5. Quick Check:

   All bigrams found, multiply 0.3*0.4=0.12 [OK]

Hint: Check if all keys exist before dictionary access [OK]

Common Mistakes:

• Assuming first word needs separate probability
• Confusing KeyError with IndexError
• Ignoring dictionary key structure

5. You want to build a trigram language model to predict the next word given two previous words. Which approach best handles the problem of unseen trigrams in your training data?

hard

A. Only use unigram probabilities for all predictions

B. Ignore unseen trigrams and assign zero probability

C. Use smoothing techniques like Kneser-Ney smoothing

D. Increase the training data size without smoothing

Solution

1. Step 1: Understand the unseen trigram problem

   Unseen trigrams cause zero probabilities, which harm model predictions.

2. Step 2: Identify solution to zero probability issue

   Smoothing techniques like Kneser-Ney adjust probabilities to handle unseen cases effectively.

3. Step 3: Evaluate other options

   Ignoring unseen trigrams or only using unigram probabilities lose context; increasing data alone may not solve sparsity.

4. Final Answer:

   Use smoothing techniques like Kneser-Ney smoothing -> Option C

5. Quick Check:

   Smoothing fixes zero probs for unseen trigrams [OK]

Hint: Use smoothing to avoid zero probabilities [OK]

Common Mistakes:

• Assigning zero probability to unseen trigrams
• Ignoring context by using only unigrams
• Relying solely on more data without smoothing