Handling out-of-vocabulary words in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of using an UNK token in NLP models?

In natural language processing, when a word is not found in the model's vocabulary, an UNK token is often used. What is the main purpose of this token?

A To remove unknown words from the input text before processing to avoid errors.

B To replace unknown words with their synonyms from a dictionary.

C To increase the vocabulary size dynamically during model training.

D To represent all out-of-vocabulary words with a single placeholder so the model can handle unknown words during inference.

💡 Hint

Think about how the model deals with words it has never seen before.

The UNK token acts as a catch-all for any word not in the vocabulary, allowing the model to process inputs without crashing or ignoring unknown words.

❓ Predict Output

intermediate

What is the output of this tokenization with OOV handling?

Given the vocabulary {'hello': 1, 'world': 2} and the sentence 'hello there world', what is the tokenized output using 0 as the UNK token index?

NLP

vocab = {'hello': 1, 'world': 2}
sentence = 'hello there world'
tokens = [vocab.get(word, 0) for word in sentence.split()]
print(tokens)

A [1, 1, 2]

B [1, 0, 2]

C [0, 1, 2]

D [1, 2, 0]

💡 Hint

Words not in the vocabulary get the UNK token index 0.

The word 'there' is not in the vocabulary, so it is replaced by 0. 'hello' maps to 1 and 'world' maps to 2.

❓ Model Choice

advanced

Which model architecture is best suited to handle out-of-vocabulary words using subword units?

You want to build an NLP model that can handle out-of-vocabulary words effectively by breaking words into smaller parts. Which model architecture or technique is best for this?

A A model using Byte Pair Encoding (BPE) or WordPiece tokenization to split words into subword units.

B A model that uses a fixed vocabulary of whole words only.

C A model that replaces all unknown words with a single UNK token without subword splitting.

D A model that ignores any word not in the vocabulary during training and inference.

💡 Hint

Think about how breaking words into smaller parts helps with unknown words.

Subword tokenization methods like BPE or WordPiece split rare or unknown words into known smaller units, allowing the model to understand and generate words it has never seen before.

❓ Hyperparameter

advanced

Which hyperparameter affects the size of the vocabulary and thus the handling of OOV words in subword tokenization?

When using Byte Pair Encoding (BPE) for tokenization, which hyperparameter directly controls the vocabulary size and impacts how many out-of-vocabulary words appear?

A The batch size used during model training.

B The learning rate of the model optimizer.

C The number of merge operations performed during BPE training.

D The dropout rate applied in the model layers.

💡 Hint

Think about what controls how many subword units are created.

The number of merge operations in BPE determines how many subword units are merged, controlling vocabulary size and granularity. More merges mean larger vocabulary and fewer OOV tokens.

❓ Metrics

expert

How does the presence of many out-of-vocabulary words affect the model's perplexity on a test set?

You evaluate a language model on a test set containing many out-of-vocabulary (OOV) words. How does this typically affect the model's perplexity metric?

A Perplexity increases because the model struggles to predict unknown words, indicating worse performance.

B Perplexity becomes zero because the model replaces all OOV words with UNK tokens.

C Perplexity remains unchanged because OOV words are ignored during evaluation.

D Perplexity decreases because unknown words are easier to predict.

💡 Hint

Think about how unknown words affect prediction confidence.

When many OOV words appear, the model cannot predict them well, leading to higher uncertainty and thus higher perplexity values, which means worse model performance.