Vocabulary size control in NLP - Model Pipeline Trace

Model Pipeline - Vocabulary size control

This pipeline shows how controlling vocabulary size helps manage text data for machine learning. It reduces the number of unique words to focus on the most important ones, making models faster and simpler.

Data Flow - 5 Stages

1Raw Text Input

1000 sentences x variable length→Collect raw text data from documents→1000 sentences x variable length

"I love apples", "Machine learning is fun"

↓

2Tokenization

1000 sentences x variable length→Split sentences into words (tokens)→1000 sentences x variable length tokens

["I", "love", "apples"], ["Machine", "learning", "is", "fun"]

↓

3Build Vocabulary

1000 sentences x variable length tokens→Count unique words and their frequencies→Vocabulary dictionary with word counts

{"I": 50, "love": 30, "apples": 20, "Machine": 40, "learning": 40, "is": 60, "fun": 25}

↓

4Vocabulary Size Control

Vocabulary dictionary with 5000 unique words→Keep top 1000 most frequent words, replace others with <UNK>→Vocabulary dictionary with 1000 words + <UNK>

Top words: {"I", "is", "Machine", ...}, others replaced by <UNK>

↓

5Text to Indexed Tokens

1000 sentences x variable length tokens→Replace words with their index in controlled vocabulary→1000 sentences x variable length indices

[1, 5, 20], [100, 200, 3, 15]

Training Trace - Epoch by Epoch

Loss
1.0 |          *
0.8 |         **
0.6 |        ***
0.4 |       ****
0.2 |      *****
    +----------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.60	Model starts learning with controlled vocabulary
2	0.65	0.72	Loss decreases and accuracy improves as model learns
3	0.50	0.80	Model converges well with reduced vocabulary size
4	0.45	0.83	Further improvement, stable training
5	0.42	0.85	Training converged with good accuracy

Prediction Trace - 4 Layers

Layer 1: Input Sentence

Layer 2: Vocabulary Mapping

Layer 3: Model Input Layer

Layer 4: Prediction Output

Model Quiz - 3 Questions

Why do we limit vocabulary size in text processing?

A To increase the number of unique words

B To reduce model complexity and focus on important words

C To make sentences longer

D To remove all rare words permanently

Key Insight

Controlling vocabulary size helps the model focus on the most frequent and important words. This reduces complexity and speeds up learning, leading to better and faster training convergence.