Tokenization and vocabulary in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to split the sentence into words using whitespace.

Prompt Engineering / GenAI

tokens = sentence.[1]()

Drag options to blanks, or click blank then click option'

Astrip

Bjoin

Csplit

Dreplace

Attempts:

3 left

💡 Hint

Common Mistakes

Using join() which combines words instead of splitting.

Using replace() which changes characters but does not split.

Using strip() which removes spaces only at the ends.

The split() method breaks a string into a list of words using whitespace by default.

2fill in blank

medium

Complete the code to create a vocabulary set from the list of tokens.

Prompt Engineering / GenAI

vocab = set([1])

Drag options to blanks, or click blank then click option'

Asentence

Btokens

Cvocab

Dwords

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the original sentence string instead of the token list.

Passing the vocabulary variable itself which is not defined yet.

Passing an undefined variable like words.

The set() function creates a collection of unique items from the list tokens.

3fill in blank

hard

Fix the error in the code to count the frequency of each token.

Prompt Engineering / GenAI

freq = {}
for token in tokens:
    freq[token] = freq.get([1], 0) + 1

Drag options to blanks, or click blank then click option'

Atoken

Btokens

Cfreq

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using the whole list tokens as the key.

Using the dictionary freq as the key.

Using an undefined variable like count.

The get() method needs the current token as the key to get its count.

4fill in blank

hard

Fill both blanks to create a dictionary of token lengths for tokens longer than 3 characters.

Prompt Engineering / GenAI

lengths = {token: [1] for token in tokens if len(token) [2] 3}

Drag options to blanks, or click blank then click option'

Alen(token)

B>

C<

Dtoken

Attempts:

3 left

💡 Hint

Common Mistakes

Using the token itself as the value instead of its length.

Using less than (<) instead of greater than (>) in the condition.

The dictionary comprehension maps each token to its length if the token length is greater than 3.

5fill in blank

hard

Fill all three blanks to create a frequency dictionary for tokens longer than 2 characters.

Prompt Engineering / GenAI

freq_filtered = {token: [1] for token in tokens if len(token) [2] 2 and token in [3]

Drag options to blanks, or click blank then click option'

Afreq[token]

B>

Cvocab

Dtokens

Attempts:

3 left

💡 Hint

Common Mistakes

Using the list of tokens instead of the vocabulary set for membership check.

Using less than (<) instead of greater than (>) in the length condition.

Using the token itself instead of its frequency as the value.

This comprehension creates a dictionary with tokens as keys and their frequencies as values, filtering tokens longer than 2 characters and present in the vocabulary set.

Practice

1. What does tokenization do in natural language processing?

easy

A. Converts tokens into images

B. Breaks text into smaller pieces called tokens

C. Removes all punctuation from text

D. Combines multiple texts into one

Solution

1. Step 1: Understand the role of tokenization

   Tokenization splits text into smaller parts called tokens, like words or subwords.

2. Step 2: Compare options with tokenization definition

   Only Breaks text into smaller pieces called tokens correctly describes breaking text into tokens.

3. Final Answer:

   Breaks text into smaller pieces called tokens -> Option B

4. Quick Check:

   Tokenization = splitting text [OK]

Hint: Tokenization means splitting text into pieces [OK]

Common Mistakes:

• Thinking tokenization changes text to images
• Confusing tokenization with removing punctuation
• Believing tokenization merges texts

2. Which of the following is the correct way to represent a token ID in Python?

easy

A. token_id = 'word'

B. token_id = {word: 1}

C. token_id = [word]

D. token_id = 123

Solution

1. Step 1: Understand token ID representation

   Token IDs are numbers representing tokens, so they should be integers.

2. Step 2: Check each option's type

   token_id = 123 assigns an integer 123, which is correct. Others use strings, lists, or dictionaries incorrectly.

3. Final Answer:

   token_id = 123 -> Option D

4. Quick Check:

   Token ID = number [OK]

Hint: Token IDs are numbers, not words or lists [OK]

Common Mistakes:

• Using strings instead of numbers for token IDs
• Confusing token IDs with token text
• Using lists or dictionaries wrongly

3. Given the vocabulary {'hello': 1, 'world': 2, '!': 3}, what is the token ID list for the text 'hello world!'?

medium

A. [1, 2, 3]

B. [0, 1, 2]

C. ['hello', 'world', '!']

D. [3, 2, 1]

Solution

1. Step 1: Map each word to its token ID

   'hello' maps to 1, 'world' maps to 2, and '!' maps to 3 according to the vocabulary.

2. Step 2: Create the token ID list in order

   The text 'hello world!' becomes [1, 2, 3].

3. Final Answer:

   [1, 2, 3] -> Option A

4. Quick Check:

   Text tokens = [1, 2, 3] [OK]

Hint: Match words to IDs in order [OK]

Common Mistakes:

• Mixing up token order
• Using token text instead of IDs
• Assigning wrong IDs from vocabulary

4. What is wrong with this tokenization code snippet?

vocab = {'hi': 1, 'there': 2}
text = 'hi there'
tokens = [vocab[word] for word in text.split() if word in vocab]

medium

A. It will raise a KeyError if a word is missing

B. It correctly tokenizes the text

C. It ignores words not in vocabulary

D. It uses split() incorrectly on the text

Solution

1. Step 1: Analyze the list comprehension

   The code splits text and includes only words found in vocab, skipping others.

2. Step 2: Identify behavior on unknown words

   Words not in vocab are ignored, which may lose information.

3. Final Answer:

   It ignores words not in vocabulary -> Option C

4. Quick Check:

   Unknown words skipped = ignoring [OK]

Hint: Check if unknown words are skipped or cause errors [OK]

Common Mistakes:

• Assuming KeyError will happen due to 'if' check
• Thinking split() is wrong here
• Missing that unknown words are ignored silently

5. You have a vocabulary with tokens: {'I':1, 'love':2, 'AI':3, '.':4}. How would you tokenize the sentence 'I love AI!' considering the exclamation mark is not in the vocabulary?

hard

A. Add '!' to vocabulary with new ID and tokenize as [1, 2, 3, 5]

B. Replace '!' with '.' and tokenize as [1, 2, 3, 4]

C. Ignore '!' and tokenize as [1, 2, 3]

D. Raise an error because '!' is unknown

Solution

1. Step 1: Understand vocabulary coverage

   The vocabulary lacks '!', so it must be added to handle the sentence fully.

2. Step 2: Add '!' with a new token ID

   Assign '!' a new ID (e.g., 5) and tokenize the sentence as [1, 2, 3, 5].

3. Final Answer:

   Add '!' to vocabulary with new ID and tokenize as [1, 2, 3, 5] -> Option A

4. Quick Check:

   Unknown token added = new ID [OK]

Hint: Add unknown tokens to vocabulary before tokenizing [OK]

Common Mistakes:

• Ignoring unknown tokens silently
• Replacing unknown tokens incorrectly
• Assuming error without handling unknown tokens