Text tokenization (Tokenizer) in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of Tokenizer on Simple Text

What is the output of the following code snippet using TensorFlow's Tokenizer?

TensorFlow

from tensorflow.keras.preprocessing.text import Tokenizer
texts = ['I love machine learning', 'Machine learning is fun']
tokenizer = Tokenizer(num_words=5)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
print(sequences)

A [[1, 2, 3, 4], [3, 4, 5, 7]]

B [[1, 2, 3, 4], [3, 4]]

C [[1, 2, 3, 4], [3, 4, 5, 1]]

D [[1, 2, 3, 4], [3, 4, 5, 6]]

💡 Hint

Remember that num_words limits the vocabulary size and words outside are ignored.

The tokenizer assigns indexes based on word frequency. Since num_words=5, only top 4 words are kept (indexes 1 to 4). Words outside this range are ignored in sequences.

❓ Model Choice

intermediate

Choosing Tokenizer Parameters for Vocabulary Size

You want to limit your tokenizer vocabulary to the 10 most frequent words in your dataset. Which parameter setting achieves this?

A Tokenizer(num_words=11)

B Tokenizer(num_words=9)

C Tokenizer(num_words=10)

D Tokenizer(num_words=0)

💡 Hint

num_words includes the top words starting from index 1.

Setting num_words=10 keeps the top 9 words plus index 0 is reserved, so effectively 9 words are kept.

❓ Metrics

advanced

Calculating Out-of-Vocabulary Token Count

Given the tokenizer below, how many tokens in the sequences are replaced by the out-of-vocabulary token?

TensorFlow

from tensorflow.keras.preprocessing.text import Tokenizer
texts = ['apple banana orange', 'banana kiwi mango']
tokenizer = Tokenizer(num_words=4, oov_token='<OOV>')
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
flat_tokens = [token for seq in sequences for token in seq]
oov_count = flat_tokens.count(tokenizer.word_index['<OOV>'])
print(oov_count)

A 2

B 1

C 3

D 0

💡 Hint

Check which words are outside the top 3 words (num_words=4 means top 3 plus OOV).

The tokenizer keeps top 3 words plus OOV token. Words 'kiwi' and 'mango' are out-of-vocabulary and replaced by OOV token, so count is 2.

🔧 Debug

advanced

Debugging Tokenizer Sequence Length Mismatch

You run the code below but get sequences of different lengths. What is the best way to fix this so all sequences have the same length?

TensorFlow

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
texts = ['I love AI', 'AI is amazing and fun']
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
print(sequences)

A Remove shorter sequences from texts before tokenizing

B Set tokenizer.num_words to length of longest sequence

C Use tokenizer.texts_to_matrix instead of texts_to_sequences

D Use pad_sequences(sequences, padding='post') to make all sequences same length

💡 Hint

Sequences of different lengths are common; padding fixes this for model input.

pad_sequences adds zeros to shorter sequences so all have equal length, which is required for batch processing in models.

🧠 Conceptual

expert

Understanding Tokenizer's word_index Behavior

After fitting a Tokenizer on texts, what does the word_index dictionary represent?

A A mapping from words to unique integer indices based on frequency

B A mapping from words to their frequency counts in the dataset

C A mapping from integer indices to words sorted alphabetically

D A mapping from words to their length in characters

💡 Hint

Think about how Tokenizer converts words to numbers for models.

word_index maps each word to a unique integer index, assigned by descending frequency order.

Practice

1. What is the main purpose of using a Tokenizer in TensorFlow text processing?

easy

A. To convert text into sequences of numbers representing words

B. To train a neural network model directly

C. To visualize text data as images

D. To generate random text samples

Solution

1. Step 1: Understand Tokenizer's role

   A Tokenizer converts words into numbers so models can understand text.

2. Step 2: Identify correct purpose

   It does not train models or create images; it prepares text data by numbering words.

3. Final Answer:

   To convert text into sequences of numbers representing words -> Option A

4. Quick Check:

   Tokenizer = text to numbers [OK]

Hint: Tokenizer turns words into numbers for models [OK]

Common Mistakes:

• Thinking Tokenizer trains the model
• Confusing tokenization with image processing
• Assuming Tokenizer generates text

2. Which of the following is the correct way to create a Tokenizer in TensorFlow?

easy

A. tokenizer = tf.keras.Tokenizer()

B. tokenizer = tf.text.Tokenizer()

C. tokenizer = tf.keras.preprocessing.text.Tokenizer()

D. tokenizer = tf.preprocessing.text.Tokenizer()

Solution

1. Step 1: Recall TensorFlow Tokenizer import path

   The Tokenizer class is in tf.keras.preprocessing.text.

2. Step 2: Match correct syntax

   tokenizer = tf.keras.preprocessing.text.Tokenizer() matches the correct full path and syntax to create a Tokenizer.

3. Final Answer:

   tokenizer = tf.keras.preprocessing.text.Tokenizer() -> Option C

4. Quick Check:

   Correct import path = tokenizer = tf.keras.preprocessing.text.Tokenizer() [OK]

Hint: Tokenizer is under tf.keras.preprocessing.text [OK]

Common Mistakes:

• Using tf.text instead of tf.keras.preprocessing.text
• Missing 'preprocessing' in the path
• Calling Tokenizer directly from tf.keras

3. Given the code:

from tensorflow.keras.preprocessing.text import Tokenizer
texts = ['hello world', 'hello tensorflow']
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(['hello tensorflow world'])
print(sequences)

What is the output?

medium

A. [[2, 1, 3]]

B. [[1, 3, 2]]

C. [[1, 2, 3]]

D. [[3, 1, 2]]

Solution

1. Step 1: Understand word index assignment

   Tokenizer assigns indexes by frequency descending, ties by order of appearance: 'hello' (2 times) = 1, 'world' (1 time, first) = 2, 'tensorflow' (1 time) = 3.

2. Step 2: Convert text to sequence

   Input 'hello tensorflow world' maps to [1, 3, 2] based on the word index.

3. Final Answer:

   [[1, 3, 2]] -> Option B

4. Quick Check:

   Word order preserved, indexes assigned by frequency then appearance [OK]

Hint: Tokenizer indexes by frequency, then maps words [OK]

Common Mistakes:

• Assuming index order is alphabetical
• Mixing word indexes
• Forgetting Tokenizer fits before sequences

4. What is wrong with this code snippet?

from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer()
texts = ['cat dog', 'dog mouse']
sequences = tokenizer.texts_to_sequences(texts)
print(sequences)

medium

A. Tokenizer is not imported correctly

B. texts variable is not a list

C. texts_to_sequences expects a list of integers

D. Tokenizer is used before fitting on texts

Solution

1. Step 1: Check Tokenizer usage order

   Tokenizer must be fitted on texts before converting texts to sequences.

2. Step 2: Identify missing fit_on_texts call

   The code misses tokenizer.fit_on_texts(texts) before texts_to_sequences.

3. Final Answer:

   Tokenizer is used before fitting on texts -> Option D

4. Quick Check:

   Fit before sequence conversion [OK]

Hint: Always call fit_on_texts before texts_to_sequences [OK]

Common Mistakes:

• Skipping fit_on_texts call
• Confusing input types for texts_to_sequences
• Assuming texts_to_sequences fits automatically

5. You want to limit your Tokenizer to only the top 3 most frequent words in your texts. Which code correctly sets this limit and converts the text 'apple banana apple orange' to sequences?

hard

A. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

B. tokenizer = Tokenizer(num_words=4)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

C. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

D. tokenizer = Tokenizer(num_words=2)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

Solution

1. Step 1: Understand num_words parameter

   Setting num_words=3 limits tokenizer to top 3 frequent words.

2. Step 2: Fit on full text and convert sequences

   Fitting on the full text with 'apple' (2 times), 'banana' (1), 'orange' (1) ranks 'apple', 'banana', 'orange' as top 3.

3. Step 3: Check output sequence

   Sequence maps 'apple' to 1, 'banana' to 2, 'orange' to 3; output is [[1, 2, 1, 3]].

4. Final Answer:

   tokenizer = Tokenizer(num_words=3) tokenizer.fit_on_texts(['apple banana apple orange']) print(tokenizer.texts_to_sequences(['apple banana apple orange'])) -> Option A

5. Quick Check:

   num_words limits vocab size correctly [OK]

Hint: Use num_words to limit vocabulary size [OK]

Common Mistakes:

• Setting num_words too low or too high
• Fitting on incomplete text
• Confusing num_words with maxlen