Language modeling concept in NLP

Introduction

Language modeling helps computers understand and predict words in sentences. It makes machines better at reading and writing like humans.

When building a chatbot that talks naturally with people.

When creating a system to autocomplete your sentences on your phone.

When translating text from one language to another.

When summarizing long articles into short paragraphs.

When detecting if a sentence makes sense or is gibberish.

Syntax

NLP

model = LanguageModel()
model.train(text_data)
prediction = model.predict(next_word | previous_words)

Language models learn from lots of text to guess what word comes next.

They can be simple (counting words) or complex (using neural networks).

Examples

This model guesses words based only on how often they appear.

NLP

Unigram model: P(word) = count(word) / total_words

This model guesses the next word based on the previous word.

NLP

Bigram model: P(word2 | word1) = count(word1, word2) / count(word1)

This model learns complex patterns and context in sentences.

NLP

Neural language model: uses a neural network to predict next word from previous words.

Sample Model

This code trains a simple neural language model to predict the next word from a small set of words. It shows how the model learns and then predicts the next word after 'hello'.

NLP

import torch
import torch.nn as nn
import torch.optim as optim

# Simple neural language model for predicting next word from a small vocabulary
class SimpleLanguageModel(nn.Module):
    def __init__(self, vocab_size, embed_dim):
        super(SimpleLanguageModel, self).__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.linear = nn.Linear(embed_dim, vocab_size)

    def forward(self, x):
        embeds = self.embedding(x)
        out = self.linear(embeds)
        return out

# Vocabulary and data
vocab = ['hello', 'world', 'good', 'morning']
word_to_ix = {w: i for i, w in enumerate(vocab)}

# Training data: pairs of (input_word, target_word)
data = [
    ('hello', 'world'),
    ('good', 'morning'),
    ('hello', 'good'),
    ('morning', 'world')
]

# Prepare inputs and targets
inputs = torch.tensor([word_to_ix[w[0]] for w in data], dtype=torch.long)
targets = torch.tensor([word_to_ix[w[1]] for w in data], dtype=torch.long)

# Model, loss, optimizer
model = SimpleLanguageModel(vocab_size=len(vocab), embed_dim=5)
loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Train for 100 epochs
for epoch in range(100):
    model.train()
    optimizer.zero_grad()
    output = model(inputs)
    loss = loss_function(output, targets)
    loss.backward()
    optimizer.step()

# Test prediction for input 'hello'
model.eval()
with torch.no_grad():
    input_word = torch.tensor([word_to_ix['hello']], dtype=torch.long)
    output = model(input_word)
    predicted_ix = torch.argmax(output, dim=1).item()
    predicted_word = vocab[predicted_ix]

print(f"Input word: 'hello'")
print(f"Predicted next word: '{predicted_word}'")
print(f"Final training loss: {loss.item():.4f}")

Important Notes

Language models get better with more text and bigger vocabularies.

Simple models like unigram ignore context, while neural models understand it better.

Training takes time and needs good examples to learn well.

Summary

Language modeling helps predict words to make computers understand text.

Models range from simple counting to complex neural networks.

They are useful in chatbots, translation, autocomplete, and more.

Practice

1. What is the main goal of a language model in natural language processing?

easy

A. To predict the next word in a sentence

B. To translate text from one language to another

C. To count the number of words in a document

D. To summarize long paragraphs into short sentences

Solution

1. Step 1: Understand the purpose of language models

   Language models are designed to understand and predict text sequences.

2. Step 2: Identify the main task of language models

   The core task is to predict the next word based on previous words in a sentence.

3. Final Answer:

   To predict the next word in a sentence -> Option A

4. Quick Check:

   Language model goal = predict next word [OK]

Hint: Language models guess the next word in text [OK]

Common Mistakes:

• Confusing language modeling with translation
• Thinking language models only count words
• Assuming summarization is the main task

2. Which of the following is the correct way to represent a bigram language model probability for a sentence "I love AI"?

easy

A. P(I) * P(love) * P(AI)

B. P(I | AI) * P(love | I) * P(AI | love)

C. P(I | love) * P(love | AI) * P(AI)

D. P(I) * P(love | I) * P(AI | love)

Solution

1. Step 1: Recall bigram model definition

   A bigram model predicts each word based on the previous word, so probabilities are conditional.

2. Step 2: Apply bigram probabilities to the sentence

   The sentence probability is P(I) * P(love | I) * P(AI | love), starting with the first word's probability.

3. Final Answer:

   P(I) * P(love | I) * P(AI | love) -> Option D

4. Quick Check:

   Bigram = word depends on previous word [OK]

Hint: Bigram means each word depends on the one before [OK]

Common Mistakes:

• Multiplying independent word probabilities (unigram)
• Using wrong conditional order
• Confusing bigram with trigram or other models

3. Given the following unigram probabilities: P(I)=0.2, P(love)=0.1, P(AI)=0.05, what is the probability of the sentence "I love AI" under a unigram model?

medium

A. 0.01

B. 0.001

C. 0.35

D. 0.0001

Solution

1. Step 1: Understand unigram model calculation

   Unigram model assumes words are independent, so multiply their probabilities.

2. Step 2: Calculate sentence probability

   Multiply P(I) * P(love) * P(AI) = 0.2 * 0.1 * 0.05 = 0.001

3. Final Answer:

   0.001 -> Option B

4. Quick Check:

   Unigram multiply all word probs = 0.001 [OK]

Hint: Multiply all word probabilities for unigram [OK]

Common Mistakes:

• Adding probabilities instead of multiplying
• Using conditional probabilities (bigram) by mistake
• Incorrect multiplication order

4. Consider this Python code snippet for a bigram model probability calculation:

sentence = ['I', 'love', 'AI']
bigram_probs = {('I', 'love'): 0.3, ('love', 'AI'): 0.4}
prob = 1.0
for i in range(len(sentence)-1):
    prob *= bigram_probs[(sentence[i], sentence[i+1])]
print(prob)

What error will occur when running this code?

medium

A. No error, prints 0.12

B. TypeError due to wrong data type in multiplication

C. KeyError because the first word probability is missing

D. IndexError because of range length

Solution

1. Step 1: Analyze the loop and dictionary access

   The loop multiplies probabilities for bigrams in the sentence using bigram_probs dictionary keys.

2. Step 2: Check if all bigrams exist in dictionary

   bigram_probs lacks a probability for the first word alone, but code only uses pairs, so no missing keys for pairs.

3. Step 3: Re-examine the code logic

   All bigrams ('I','love') and ('love','AI') exist in dictionary, so no KeyError. No TypeError or IndexError expected.

4. Final Answer:

   No error, prints 0.12 -> Option A

5. Quick Check:

   All bigrams found, multiply 0.3*0.4=0.12 [OK]

Hint: Check if all keys exist before dictionary access [OK]

Common Mistakes:

• Assuming first word needs separate probability
• Confusing KeyError with IndexError
• Ignoring dictionary key structure

5. You want to build a trigram language model to predict the next word given two previous words. Which approach best handles the problem of unseen trigrams in your training data?

hard

A. Only use unigram probabilities for all predictions

B. Ignore unseen trigrams and assign zero probability

C. Use smoothing techniques like Kneser-Ney smoothing

D. Increase the training data size without smoothing

Solution

1. Step 1: Understand the unseen trigram problem

   Unseen trigrams cause zero probabilities, which harm model predictions.

2. Step 2: Identify solution to zero probability issue

   Smoothing techniques like Kneser-Ney adjust probabilities to handle unseen cases effectively.

3. Step 3: Evaluate other options

   Ignoring unseen trigrams or only using unigram probabilities lose context; increasing data alone may not solve sparsity.

4. Final Answer:

   Use smoothing techniques like Kneser-Ney smoothing -> Option C

5. Quick Check:

   Smoothing fixes zero probs for unseen trigrams [OK]

Hint: Use smoothing to avoid zero probabilities [OK]

Common Mistakes:

• Assigning zero probability to unseen trigrams
• Ignoring context by using only unigrams
• Relying solely on more data without smoothing