Vocabulary size control in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why limit vocabulary size in NLP models?

In natural language processing, why do we often limit the vocabulary size when building models?

A To reduce model complexity and memory usage while focusing on the most frequent words

B To make the model ignore common words like 'the' and 'and'

C To increase the number of rare words the model can learn

D To ensure the model only learns from stop words

💡 Hint

Think about how large vocabularies affect model size and training speed.

Limiting vocabulary size helps keep the model smaller and faster by focusing on frequent words, which carry most meaning. Rare words are often replaced with a special token.

❓ Predict Output

intermediate

Output of vocabulary size after filtering

Given the code below that filters tokens by frequency, what is the length of the resulting vocabulary?

NLP

from collections import Counter

texts = ['apple banana apple', 'banana orange apple', 'orange banana banana']
tokens = ' '.join(texts).split()
counter = Counter(tokens)
vocab = {word for word, freq in counter.items() if freq >= 3}
print(len(vocab))

A 2

B 3

C 1

D 0

💡 Hint

Count how many words appear at least 3 times.

'apple' appears 3 times, 'banana' 4 times, 'orange' 2 times. Only 'apple' and 'banana' meet the frequency threshold.

❓ Model Choice

advanced

Choosing a tokenization method to control vocabulary size

You want to control vocabulary size in a text classification task. Which tokenization method helps best to keep vocabulary size manageable while capturing meaningful subword units?

A Character-level tokenization

B Word-level tokenization without filtering

C Sentence-level tokenization

D Byte Pair Encoding (BPE) subword tokenization

💡 Hint

Think about methods that break words into smaller parts to reduce vocabulary size.

BPE breaks words into subword units, reducing vocabulary size while preserving meaning better than character-level tokenization.

❓ Metrics

advanced

Effect of vocabulary size on model perplexity

When training a language model, how does increasing vocabulary size generally affect the model's perplexity on test data?

A Perplexity always decreases as vocabulary size increases

B Perplexity may increase due to data sparsity with very large vocabularies

C Perplexity always increases as vocabulary size decreases

D Perplexity remains unchanged regardless of vocabulary size

💡 Hint

Consider the trade-off between vocabulary coverage and data sparsity.

Very large vocabularies cause many rare words, leading to sparse data and higher perplexity. Moderate vocabularies balance coverage and data density.

🔧 Debug

expert

Identifying the bug in vocabulary size control code

What error does the following code raise when trying to limit vocabulary size by frequency?

from collections import Counter
texts = ['cat dog cat', 'dog mouse cat', 'mouse dog dog']
tokens = ' '.join(texts).split()
counter = Counter(tokens)
vocab = {word for word in counter if counter[word] > 2}
print(vocab[0])

A SyntaxError: invalid syntax

B KeyError: 0

C TypeError: 'set' object is not subscriptable

D No error, prints 'cat'

💡 Hint

Look at how vocab is accessed after creation.

vocab is a set, which cannot be accessed by index like a list. Trying vocab[0] causes a TypeError.