Model Pipeline - N-gram language models
An N-gram language model predicts the next word in a sentence by looking at the previous N-1 words. It learns from text data by counting word sequences and uses these counts to guess what comes next.
0
0
N-gram language models in NLP - Model Pipeline Trace
Data Flow - 6 Stages
1Data in
10000 sentences x variable length→Raw text sentences collected for training→10000 sentences x variable length
"I love machine learning", "The cat sat on the mat"
↓
2Preprocessing
10000 sentences x variable length→Lowercase, remove punctuation, tokenize words→10000 sentences x variable length tokens
["i", "love", "machine", "learning"]
↓
3Feature Engineering
10000 sentences x variable length tokens→Extract N-grams (e.g., bigrams for N=2)→List of N-grams with counts
[('i', 'love'), ('love', 'machine'), ('machine', 'learning')]
↓
4Model Trains
N-gram counts→Calculate probabilities of next word given previous N-1 words→Probability tables for N-grams
P('learning'|'machine') = 0.8
↓
5Metrics Improve
Validation text data→Calculate perplexity to measure model quality→Perplexity score (lower is better)
Perplexity = 120
↓
6Prediction
Previous N-1 words→Use probability tables to predict next word→Next word prediction
Input: 'learning' -> Output: 'is'
Training Trace - Epoch by Epoch
5.0 |*****
4.0 |****
3.0 |***
2.0 |**
1.0 |*
+-----
1 2 3 4 5 Epochs
| Epoch | Loss ↓ | Accuracy ↑ | Observation |
|---|---|---|---|
| 1 | 5.0 | 0.10 | Initial model with random-like predictions |
| 2 | 3.8 | 0.25 | Model starts learning common word sequences |
| 3 | 2.9 | 0.40 | Better prediction of next words |
| 4 | 2.3 | 0.55 | Model captures frequent N-grams well |
| 5 | 1.9 | 0.65 | Converging with improved next word guesses |
Prediction Trace - 3 Layers
Layer 1: Input previous words
Layer 2: Look up N-gram probabilities
Layer 3: Select next word
Model Quiz - 3 Questions
Test your understanding
What does the N-gram model use to predict the next word?
Key Insight
N-gram models learn word patterns by counting sequences and estimating probabilities. As training progresses, the model better predicts the next word, shown by decreasing loss and perplexity. This simple approach captures local word context effectively.