Tokenization and vocabulary in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Tokenization Types

Which of the following best describes subword tokenization in natural language processing?

A Splitting text into individual characters only

B Breaking text into whole words without splitting

C Dividing text into smaller units than words, like syllables or parts of words

D Ignoring spaces and treating the entire text as one token

💡 Hint

Think about how tokenization helps handle unknown or rare words by breaking them down.

Subword tokenization breaks words into smaller meaningful parts, allowing models to understand rare or new words by their components.

❓ Predict Output

intermediate

Output of Tokenization Code

What is the output of the following Python code using a simple whitespace tokenizer?

Prompt Engineering / GenAI

text = "Machine learning is fun"
tokens = text.split()
print(tokens)

A ['Machine_learning_is_fun']

B ['Machine learning is fun']

C ['M', 'a', 'c', 'h', 'i', 'n', 'e']

D ['Machine', 'learning', 'is', 'fun']

💡 Hint

Remember what the split() method does by default.

The split() method without arguments splits the string at spaces, producing a list of words.

❓ Model Choice

advanced

Choosing Vocabulary Size for Tokenization

You want to train a language model on a large dataset with many rare words. Which vocabulary size is best to balance coverage and model size?

A Moderate vocabulary size (e.g., 30,000 tokens) with subword tokenization

B Very small vocabulary (e.g., 500 tokens) to reduce model size

C Very large vocabulary (e.g., 100,000 tokens) to cover all words exactly

D Vocabulary size does not affect model performance

💡 Hint

Think about how subword tokenization helps with rare words and model efficiency.

A moderate vocabulary with subword tokens balances coverage of rare words and keeps the model size manageable.

❓ Metrics

advanced

Evaluating Tokenizer Coverage

You have a tokenizer vocabulary of 10,000 tokens. After tokenizing a test set of 1,000 words, 50 words are split into multiple tokens. What is the approximate tokenization coverage percentage?

A 95%

B 5%

C 50%

D 100%

💡 Hint

Coverage means how many words are represented as single tokens.

If 50 words are split, then 950 words are covered as single tokens. Coverage = (950/1000)*100 = 95%.

🔧 Debug

expert

Identifying Tokenization Bug in Code

What error does the following code raise when trying to tokenize text using a vocabulary dictionary?

vocab = {"hello": 1, "world": 2}
text = "hello unknown world"
tokens = [vocab[word] for word in text.split()]
print(tokens)

A No error, output is [1, 0, 2]

B KeyError because 'unknown' is not in vocab

C TypeError because vocab is not iterable

D SyntaxError due to missing colon

💡 Hint

Check what happens when a word is not found in the dictionary keys.

Accessing a dictionary with a key that does not exist raises a KeyError in Python.

Practice

1. What does tokenization do in natural language processing?

easy

A. Converts tokens into images

B. Breaks text into smaller pieces called tokens

C. Removes all punctuation from text

D. Combines multiple texts into one

Solution

1. Step 1: Understand the role of tokenization

   Tokenization splits text into smaller parts called tokens, like words or subwords.

2. Step 2: Compare options with tokenization definition

   Only Breaks text into smaller pieces called tokens correctly describes breaking text into tokens.

3. Final Answer:

   Breaks text into smaller pieces called tokens -> Option B

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into pieces [OK]

Common Mistakes:

• Thinking tokenization changes text to images
• Confusing tokenization with removing punctuation
• Believing tokenization merges texts

2. Which of the following is the correct way to represent a token ID in Python?

easy

A. token_id = 'word'

B. token_id = {word: 1}

C. token_id = [word]

D. token_id = 123

Solution

1. Step 1: Understand token ID representation

   Token IDs are numbers representing tokens, so they should be integers.

2. Step 2: Check each option's type

   token_id = 123 assigns an integer 123, which is correct. Others use strings, lists, or dictionaries incorrectly.

3. Final Answer:

   token_id = 123 -> Option D

4. Quick Check:

   Token ID = number [OK]

Hint: Token IDs are numbers, not words or lists [OK]

Common Mistakes:

• Using strings instead of numbers for token IDs
• Confusing token IDs with token text
• Using lists or dictionaries wrongly

3. Given the vocabulary {'hello': 1, 'world': 2, '!': 3}, what is the token ID list for the text 'hello world!'?

medium

A. [1, 2, 3]

B. [0, 1, 2]

C. ['hello', 'world', '!']

D. [3, 2, 1]

Solution

1. Step 1: Map each word to its token ID

   'hello' maps to 1, 'world' maps to 2, and '!' maps to 3 according to the vocabulary.

2. Step 2: Create the token ID list in order

   The text 'hello world!' becomes [1, 2, 3].

3. Final Answer:

   [1, 2, 3] -> Option A

4. Quick Check:

   Text tokens = [1, 2, 3] [OK]

Hint: Match words to IDs in order [OK]

Common Mistakes:

• Mixing up token order
• Using token text instead of IDs
• Assigning wrong IDs from vocabulary

4. What is wrong with this tokenization code snippet?

vocab = {'hi': 1, 'there': 2}
text = 'hi there'
tokens = [vocab[word] for word in text.split() if word in vocab]

medium

A. It will raise a KeyError if a word is missing

B. It correctly tokenizes the text

C. It ignores words not in vocabulary

D. It uses split() incorrectly on the text

Solution

1. Step 1: Analyze the list comprehension

   The code splits text and includes only words found in vocab, skipping others.

2. Step 2: Identify behavior on unknown words

   Words not in vocab are ignored, which may lose information.

3. Final Answer:

   It ignores words not in vocabulary -> Option C

4. Quick Check:

   Unknown words skipped = ignoring [OK]

Hint: Check if unknown words are skipped or cause errors [OK]

Common Mistakes:

• Assuming KeyError will happen due to 'if' check
• Thinking split() is wrong here
• Missing that unknown words are ignored silently

5. You have a vocabulary with tokens: {'I':1, 'love':2, 'AI':3, '.':4}. How would you tokenize the sentence 'I love AI!' considering the exclamation mark is not in the vocabulary?

hard

A. Add '!' to vocabulary with new ID and tokenize as [1, 2, 3, 5]

B. Replace '!' with '.' and tokenize as [1, 2, 3, 4]

C. Ignore '!' and tokenize as [1, 2, 3]

D. Raise an error because '!' is unknown

Solution

1. Step 1: Understand vocabulary coverage

   The vocabulary lacks '!', so it must be added to handle the sentence fully.

2. Step 2: Add '!' with a new token ID

   Assign '!' a new ID (e.g., 5) and tokenize the sentence as [1, 2, 3, 5].

3. Final Answer:

   Add '!' to vocabulary with new ID and tokenize as [1, 2, 3, 5] -> Option A

4. Quick Check:

   Unknown token added = new ID [OK]

Hint: Add unknown tokens to vocabulary before tokenizing [OK]

Common Mistakes:

• Ignoring unknown tokens silently
• Replacing unknown tokens incorrectly
• Assuming error without handling unknown tokens