Text tokenization (Tokenizer) in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a tokenizer that splits text into words.

TensorFlow

from tensorflow.keras.preprocessing.text import Tokenizer

texts = ['I love machine learning']
tokenizer = Tokenizer(num_words=100)
tokenizer.[1](texts)

Drag options to blanks, or click blank then click option'

Atokenize

Btexts_to_sequences

Csequences_to_texts

Dfit_on_texts

Attempts:

3 left

💡 Hint

Common Mistakes

Using texts_to_sequences before fitting the tokenizer.

Trying to call a non-existent method 'tokenize'.

The method fit_on_texts() trains the tokenizer on the given texts to build the word index.

2fill in blank

medium

Complete the code to convert texts into sequences of integers.

TensorFlow

sequences = tokenizer.[1](texts)

Drag options to blanks, or click blank then click option'

Atexts_to_sequences

Bsequences_to_texts

Cfit_on_texts

Dtokenize

Attempts:

3 left

💡 Hint

Common Mistakes

Calling fit_on_texts again instead of converting texts.

Using sequences_to_texts which does the opposite.

texts_to_sequences() converts texts into lists of integer tokens based on the tokenizer's word index.

3fill in blank

hard

Fix the error in the code to get the word index dictionary.

TensorFlow

word_index = tokenizer.[1]

Drag options to blanks, or click blank then click option'

Aword_index()

Bword_index

Cget_word_index()

Dget_word_index

Attempts:

3 left

💡 Hint

Common Mistakes

Adding parentheses after word_index causing a TypeError.

Trying to call a non-existent method get_word_index.

word_index is an attribute, not a method, so no parentheses are needed.

4fill in blank

hard

Fill both blanks to create a dictionary of word counts from texts.

TensorFlow

word_counts = tokenizer.[1]
print(word_counts.[2]('machine'))

Drag options to blanks, or click blank then click option'

Aword_counts

Bword_index

Cget

Dcount

Attempts:

3 left

💡 Hint

Common Mistakes

Using word_index instead of word_counts for counts.

Using count() which is not a dictionary method.

word_counts is an attribute holding word frequencies; get() retrieves the count for a given word safely.

5fill in blank

hard

Fill all three blanks to create sequences and pad them to length 5.

TensorFlow

from tensorflow.keras.preprocessing.sequence import pad_sequences

sequences = tokenizer.[1](texts)
padded = pad_sequences(sequences, maxlen=[2], padding=[3])
print(padded)

Drag options to blanks, or click blank then click option'

Atexts_to_sequences

B3

C'post'

D'pre'

Attempts:

3 left

💡 Hint

Common Mistakes

Using fit_on_texts instead of texts_to_sequences for sequences.

Setting maxlen too large or too small.

Using 'post' padding instead of 'pre'.

texts_to_sequences converts texts to sequences; maxlen=3 sets length; padding='pre' adds zeros at start.

Practice

1. What is the main purpose of using a Tokenizer in TensorFlow text processing?

easy

A. To convert text into sequences of numbers representing words

B. To train a neural network model directly

C. To visualize text data as images

D. To generate random text samples

Solution

1. Step 1: Understand Tokenizer's role

   A Tokenizer converts words into numbers so models can understand text.

2. Step 2: Identify correct purpose

   It does not train models or create images; it prepares text data by numbering words.

3. Final Answer:

   To convert text into sequences of numbers representing words -> Option A

4. Quick Check:

   Tokenizer = text to numbers [OK]

Hint: Tokenizer turns words into numbers for models [OK]

Common Mistakes:

• Thinking Tokenizer trains the model
• Confusing tokenization with image processing
• Assuming Tokenizer generates text

2. Which of the following is the correct way to create a Tokenizer in TensorFlow?

easy

A. tokenizer = tf.keras.Tokenizer()

B. tokenizer = tf.text.Tokenizer()

C. tokenizer = tf.keras.preprocessing.text.Tokenizer()

D. tokenizer = tf.preprocessing.text.Tokenizer()

Solution

1. Step 1: Recall TensorFlow Tokenizer import path

   The Tokenizer class is in tf.keras.preprocessing.text.

2. Step 2: Match correct syntax

   tokenizer = tf.keras.preprocessing.text.Tokenizer() matches the correct full path and syntax to create a Tokenizer.

3. Final Answer:

   tokenizer = tf.keras.preprocessing.text.Tokenizer() -> Option C

4. Quick Check:

   Correct import path = tokenizer = tf.keras.preprocessing.text.Tokenizer() [OK]

Hint: Tokenizer is under tf.keras.preprocessing.text [OK]

Common Mistakes:

• Using tf.text instead of tf.keras.preprocessing.text
• Missing 'preprocessing' in the path
• Calling Tokenizer directly from tf.keras

3. Given the code:

from tensorflow.keras.preprocessing.text import Tokenizer
texts = ['hello world', 'hello tensorflow']
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(['hello tensorflow world'])
print(sequences)

What is the output?

medium

A. [[2, 1, 3]]

B. [[1, 3, 2]]

C. [[1, 2, 3]]

D. [[3, 1, 2]]

Solution

1. Step 1: Understand word index assignment

   Tokenizer assigns indexes by frequency descending, ties by order of appearance: 'hello' (2 times) = 1, 'world' (1 time, first) = 2, 'tensorflow' (1 time) = 3.

2. Step 2: Convert text to sequence

   Input 'hello tensorflow world' maps to [1, 3, 2] based on the word index.

3. Final Answer:

   [[1, 3, 2]] -> Option B

4. Quick Check:

   Word order preserved, indexes assigned by frequency then appearance [OK]

Hint: Tokenizer indexes by frequency, then maps words [OK]

Common Mistakes:

• Assuming index order is alphabetical
• Mixing word indexes
• Forgetting Tokenizer fits before sequences

4. What is wrong with this code snippet?

from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer()
texts = ['cat dog', 'dog mouse']
sequences = tokenizer.texts_to_sequences(texts)
print(sequences)

medium

A. Tokenizer is not imported correctly

B. texts variable is not a list

C. texts_to_sequences expects a list of integers

D. Tokenizer is used before fitting on texts

Solution

1. Step 1: Check Tokenizer usage order

   Tokenizer must be fitted on texts before converting texts to sequences.

2. Step 2: Identify missing fit_on_texts call

   The code misses tokenizer.fit_on_texts(texts) before texts_to_sequences.

3. Final Answer:

   Tokenizer is used before fitting on texts -> Option D

4. Quick Check:

   Fit before sequence conversion [OK]

Hint: Always call fit_on_texts before texts_to_sequences [OK]

Common Mistakes:

• Skipping fit_on_texts call
• Confusing input types for texts_to_sequences
• Assuming texts_to_sequences fits automatically

5. You want to limit your Tokenizer to only the top 3 most frequent words in your texts. Which code correctly sets this limit and converts the text 'apple banana apple orange' to sequences?

hard

A. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

B. tokenizer = Tokenizer(num_words=4)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

C. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

D. tokenizer = Tokenizer(num_words=2)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

Solution

1. Step 1: Understand num_words parameter

   Setting num_words=3 limits tokenizer to top 3 frequent words.

2. Step 2: Fit on full text and convert sequences

   Fitting on the full text with 'apple' (2 times), 'banana' (1), 'orange' (1) ranks 'apple', 'banana', 'orange' as top 3.

3. Step 3: Check output sequence

   Sequence maps 'apple' to 1, 'banana' to 2, 'orange' to 3; output is [[1, 2, 1, 3]].

4. Final Answer:

   tokenizer = Tokenizer(num_words=3) tokenizer.fit_on_texts(['apple banana apple orange']) print(tokenizer.texts_to_sequences(['apple banana apple orange'])) -> Option A

5. Quick Check:

   num_words limits vocab size correctly [OK]

Hint: Use num_words to limit vocabulary size [OK]

Common Mistakes:

• Setting num_words too low or too high
• Fitting on incomplete text
• Confusing num_words with maxlen