Language modeling concept in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to load a pre-trained language model using Hugging Face Transformers.

NLP

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained([1])

Drag options to blanks, or click blank then click option'

A"bert-base-uncased"

B"gpt2"

C"resnet50"

D"vgg16"

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing BERT which is not a causal language model.

Selecting image models like ResNet or VGG.

The correct model for language modeling here is "gpt2", which is a causal language model. BERT is not a causal LM, and ResNet/VGG are image models.

2fill in blank

medium

Complete the code to tokenize input text for the language model.

NLP

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer([1], return_tensors="pt")

Drag options to blanks, or click blank then click option'

A['Hello, how are you?']

B12345

C"Hello, how are you?"

DNone

Attempts:

3 left

💡 Hint

Common Mistakes

Passing a list instead of a string.

Passing a number or None.

The tokenizer expects a string input for text. Passing a string like "Hello, how are you?" is correct. Passing a list or number is incorrect here.

3fill in blank

hard

Fix the error in the code to generate text from the model.

NLP

outputs = model.generate([1])
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Drag options to blanks, or click blank then click option'

Ainputs

Binputs['token_type_ids']

Cinputs['attention_mask']

Dinputs['input_ids']

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the entire inputs dictionary instead of input_ids.

Passing attention_mask or token_type_ids instead of input_ids.

The generate method requires input IDs tensor, which is accessed by inputs['input_ids']. Passing the whole inputs dict or other keys causes errors.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps words to their lengths only if length is greater than 3.

NLP

word_lengths = {word: [1] for word in words if [2]

Drag options to blanks, or click blank then click option'

Alen(word)

Bword

Clen(word) > 3

Dword > 3

Attempts:

3 left

💡 Hint

Common Mistakes

Using the word itself as the value instead of its length.

Checking if the word string is greater than 3 instead of length.

The dictionary comprehension maps each word to its length (len(word)) only if the length is greater than 3 (len(word) > 3).

5fill in blank

hard

Fill all three blanks to create a filtered dictionary of words with length greater than 4 and convert keys to uppercase.

NLP

filtered = [1]: [2] for [3] in words if len([3]) > 4}

Drag options to blanks, or click blank then click option'

Aword.upper()

Bword

Dlen(word)

Attempts:

3 left

💡 Hint

Common Mistakes

Using different variable names inconsistently.

Not converting keys to uppercase.

The dictionary comprehension uses the uppercase word as key (word.upper()), the original word as value, and iterates over 'word' in words.

Practice

1. What is the main goal of a language model in natural language processing?

easy

A. To predict the next word in a sentence

B. To translate text from one language to another

C. To count the number of words in a document

D. To summarize long paragraphs into short sentences

Solution

1. Step 1: Understand the purpose of language models

   Language models are designed to understand and predict text sequences.

2. Step 2: Identify the main task of language models

   The core task is to predict the next word based on previous words in a sentence.

3. Final Answer:

   To predict the next word in a sentence -> Option A

4. Quick Check:

   Language model goal = predict next word [OK]

Hint: Language models guess the next word in text [OK]

Common Mistakes:

• Confusing language modeling with translation
• Thinking language models only count words
• Assuming summarization is the main task

2. Which of the following is the correct way to represent a bigram language model probability for a sentence "I love AI"?

easy

A. P(I) * P(love) * P(AI)

B. P(I | AI) * P(love | I) * P(AI | love)

C. P(I | love) * P(love | AI) * P(AI)

D. P(I) * P(love | I) * P(AI | love)

Solution

1. Step 1: Recall bigram model definition

   A bigram model predicts each word based on the previous word, so probabilities are conditional.

2. Step 2: Apply bigram probabilities to the sentence

   The sentence probability is P(I) * P(love | I) * P(AI | love), starting with the first word's probability.

3. Final Answer:

   P(I) * P(love | I) * P(AI | love) -> Option D

4. Quick Check:

   Bigram = word depends on previous word [OK]

Hint: Bigram means each word depends on the one before [OK]

Common Mistakes:

• Multiplying independent word probabilities (unigram)
• Using wrong conditional order
• Confusing bigram with trigram or other models

3. Given the following unigram probabilities: P(I)=0.2, P(love)=0.1, P(AI)=0.05, what is the probability of the sentence "I love AI" under a unigram model?

medium

A. 0.01

B. 0.001

C. 0.35

D. 0.0001

Solution

1. Step 1: Understand unigram model calculation

   Unigram model assumes words are independent, so multiply their probabilities.

2. Step 2: Calculate sentence probability

   Multiply P(I) * P(love) * P(AI) = 0.2 * 0.1 * 0.05 = 0.001

3. Final Answer:

   0.001 -> Option B

4. Quick Check:

   Unigram multiply all word probs = 0.001 [OK]

Hint: Multiply all word probabilities for unigram [OK]

Common Mistakes:

• Adding probabilities instead of multiplying
• Using conditional probabilities (bigram) by mistake
• Incorrect multiplication order

4. Consider this Python code snippet for a bigram model probability calculation:

sentence = ['I', 'love', 'AI']
bigram_probs = {('I', 'love'): 0.3, ('love', 'AI'): 0.4}
prob = 1.0
for i in range(len(sentence)-1):
    prob *= bigram_probs[(sentence[i], sentence[i+1])]
print(prob)

What error will occur when running this code?

medium

A. No error, prints 0.12

B. TypeError due to wrong data type in multiplication

C. KeyError because the first word probability is missing

D. IndexError because of range length

Solution

1. Step 1: Analyze the loop and dictionary access

   The loop multiplies probabilities for bigrams in the sentence using bigram_probs dictionary keys.

2. Step 2: Check if all bigrams exist in dictionary

   bigram_probs lacks a probability for the first word alone, but code only uses pairs, so no missing keys for pairs.

3. Step 3: Re-examine the code logic

   All bigrams ('I','love') and ('love','AI') exist in dictionary, so no KeyError. No TypeError or IndexError expected.

4. Final Answer:

   No error, prints 0.12 -> Option A

5. Quick Check:

   All bigrams found, multiply 0.3*0.4=0.12 [OK]

Hint: Check if all keys exist before dictionary access [OK]

Common Mistakes:

• Assuming first word needs separate probability
• Confusing KeyError with IndexError
• Ignoring dictionary key structure

5. You want to build a trigram language model to predict the next word given two previous words. Which approach best handles the problem of unseen trigrams in your training data?

hard

A. Only use unigram probabilities for all predictions

B. Ignore unseen trigrams and assign zero probability

C. Use smoothing techniques like Kneser-Ney smoothing

D. Increase the training data size without smoothing

Solution

1. Step 1: Understand the unseen trigram problem

   Unseen trigrams cause zero probabilities, which harm model predictions.

2. Step 2: Identify solution to zero probability issue

   Smoothing techniques like Kneser-Ney adjust probabilities to handle unseen cases effectively.

3. Step 3: Evaluate other options

   Ignoring unseen trigrams or only using unigram probabilities lose context; increasing data alone may not solve sparsity.

4. Final Answer:

   Use smoothing techniques like Kneser-Ney smoothing -> Option C

5. Quick Check:

   Smoothing fixes zero probs for unseen trigrams [OK]

Hint: Use smoothing to avoid zero probabilities [OK]

Common Mistakes:

• Assigning zero probability to unseen trigrams
• Ignoring context by using only unigrams
• Relying solely on more data without smoothing