Text tokenization (Tokenizer) in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Text tokenization (Tokenizer)

Which metric matters for Text Tokenization and WHY

For text tokenization, the key metric is tokenization accuracy. This means how well the tokenizer splits text into meaningful pieces (tokens) without losing or mixing words. Good tokenization helps the model understand text better, leading to better predictions.

We also care about vocabulary coverage, which shows how many words in the text are recognized by the tokenizer. If many words are unknown, the model may struggle.

Confusion Matrix or Equivalent Visualization

Tokenization Example:

Original text: "I love AI!"
Expected tokens: ["I", "love", "AI", "!"]
Tokenizer output: ["I", "love", "AI", "!"]

Correct tokens: 4
Incorrect tokens: 0

Tokenization Accuracy = 4 / 4 = 100%

If tokenizer output was ["I", "lo", "ve", "AI", "!"]:
Correct tokens: 3 ("I", "AI", "!")
Incorrect tokens: 2 ("lo", "ve" instead of "love")

Tokenization Accuracy = 3 / 5 = 60%
    

Precision vs Recall Tradeoff in Tokenization

In tokenization, precision means how many tokens produced are correct, and recall means how many correct tokens were found out of all possible tokens.

If tokenizer splits too much (over-segmentation), precision drops because many tokens are wrong. If tokenizer splits too little (under-segmentation), recall drops because some tokens are missed.

Example: For a chatbot, high recall is important to catch all words so the bot understands user intent fully. For a search engine, high precision is important to avoid confusing results.

What Good vs Bad Tokenization Metrics Look Like

• Good: Tokenization accuracy above 95%, high vocabulary coverage, tokens match expected words closely.
• Bad: Accuracy below 80%, many unknown tokens, tokens split words incorrectly or merge multiple words.

Good tokenization leads to better model training and predictions. Bad tokenization confuses the model and lowers performance.

Common Pitfalls in Tokenization Metrics

• Ignoring unknown tokens: Not counting how many words are not in the tokenizer's vocabulary can hide problems.
• Overfitting tokenizer: Tokenizer trained on small data may not generalize to new text.
• Confusing token count with token quality: More tokens do not always mean better understanding.
• Data leakage: Using test data to build tokenizer vocabulary can give false high accuracy.

Self Check

Your tokenizer has 98% tokenization accuracy but only 60% vocabulary coverage on new text. Is it good?

Answer: Not really. High accuracy means it splits known words well, but low vocabulary coverage means many words are unknown. The tokenizer may miss important words, hurting model understanding. You should improve vocabulary or use a tokenizer that handles unknown words better.

Key Result

Tokenization accuracy and vocabulary coverage are key to good text tokenization performance.

Practice

1. What is the main purpose of using a Tokenizer in TensorFlow text processing?

easy

A. To convert text into sequences of numbers representing words

B. To train a neural network model directly

C. To visualize text data as images

D. To generate random text samples

Solution

1. Step 1: Understand Tokenizer's role

   A Tokenizer converts words into numbers so models can understand text.

2. Step 2: Identify correct purpose

   It does not train models or create images; it prepares text data by numbering words.

3. Final Answer:

   To convert text into sequences of numbers representing words -> Option A

4. Quick Check:

   Tokenizer = text to numbers [OK]

Hint: Tokenizer turns words into numbers for models [OK]

Common Mistakes:

• Thinking Tokenizer trains the model
• Confusing tokenization with image processing
• Assuming Tokenizer generates text

2. Which of the following is the correct way to create a Tokenizer in TensorFlow?

easy

A. tokenizer = tf.keras.Tokenizer()

B. tokenizer = tf.text.Tokenizer()

C. tokenizer = tf.keras.preprocessing.text.Tokenizer()

D. tokenizer = tf.preprocessing.text.Tokenizer()

Solution

1. Step 1: Recall TensorFlow Tokenizer import path

   The Tokenizer class is in tf.keras.preprocessing.text.

2. Step 2: Match correct syntax

   tokenizer = tf.keras.preprocessing.text.Tokenizer() matches the correct full path and syntax to create a Tokenizer.

3. Final Answer:

   tokenizer = tf.keras.preprocessing.text.Tokenizer() -> Option C

4. Quick Check:

   Correct import path = tokenizer = tf.keras.preprocessing.text.Tokenizer() [OK]

Hint: Tokenizer is under tf.keras.preprocessing.text [OK]

Common Mistakes:

• Using tf.text instead of tf.keras.preprocessing.text
• Missing 'preprocessing' in the path
• Calling Tokenizer directly from tf.keras

3. Given the code:

from tensorflow.keras.preprocessing.text import Tokenizer
texts = ['hello world', 'hello tensorflow']
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(['hello tensorflow world'])
print(sequences)

What is the output?

medium

A. [[2, 1, 3]]

B. [[1, 3, 2]]

C. [[1, 2, 3]]

D. [[3, 1, 2]]

Solution

1. Step 1: Understand word index assignment

   Tokenizer assigns indexes by frequency descending, ties by order of appearance: 'hello' (2 times) = 1, 'world' (1 time, first) = 2, 'tensorflow' (1 time) = 3.

2. Step 2: Convert text to sequence

   Input 'hello tensorflow world' maps to [1, 3, 2] based on the word index.

3. Final Answer:

   [[1, 3, 2]] -> Option B

4. Quick Check:

   Word order preserved, indexes assigned by frequency then appearance [OK]

Hint: Tokenizer indexes by frequency, then maps words [OK]

Common Mistakes:

• Assuming index order is alphabetical
• Mixing word indexes
• Forgetting Tokenizer fits before sequences

4. What is wrong with this code snippet?

from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer()
texts = ['cat dog', 'dog mouse']
sequences = tokenizer.texts_to_sequences(texts)
print(sequences)

medium

A. Tokenizer is not imported correctly

B. texts variable is not a list

C. texts_to_sequences expects a list of integers

D. Tokenizer is used before fitting on texts

Solution

1. Step 1: Check Tokenizer usage order

   Tokenizer must be fitted on texts before converting texts to sequences.

2. Step 2: Identify missing fit_on_texts call

   The code misses tokenizer.fit_on_texts(texts) before texts_to_sequences.

3. Final Answer:

   Tokenizer is used before fitting on texts -> Option D

4. Quick Check:

   Fit before sequence conversion [OK]

Hint: Always call fit_on_texts before texts_to_sequences [OK]

Common Mistakes:

• Skipping fit_on_texts call
• Confusing input types for texts_to_sequences
• Assuming texts_to_sequences fits automatically

5. You want to limit your Tokenizer to only the top 3 most frequent words in your texts. Which code correctly sets this limit and converts the text 'apple banana apple orange' to sequences?

hard

A. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

B. tokenizer = Tokenizer(num_words=4)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

C. tokenizer = Tokenizer(num_words=3)
tokenizer.fit_on_texts(['apple banana orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

D. tokenizer = Tokenizer(num_words=2)
tokenizer.fit_on_texts(['apple banana apple orange'])
print(tokenizer.texts_to_sequences(['apple banana apple orange']))

Solution

1. Step 1: Understand num_words parameter

   Setting num_words=3 limits tokenizer to top 3 frequent words.

2. Step 2: Fit on full text and convert sequences

   Fitting on the full text with 'apple' (2 times), 'banana' (1), 'orange' (1) ranks 'apple', 'banana', 'orange' as top 3.

3. Step 3: Check output sequence

   Sequence maps 'apple' to 1, 'banana' to 2, 'orange' to 3; output is [[1, 2, 1, 3]].

4. Final Answer:

   tokenizer = Tokenizer(num_words=3) tokenizer.fit_on_texts(['apple banana apple orange']) print(tokenizer.texts_to_sequences(['apple banana apple orange'])) -> Option A

5. Quick Check:

   num_words limits vocab size correctly [OK]

Hint: Use num_words to limit vocabulary size [OK]

Common Mistakes:

• Setting num_words too low or too high
• Fitting on incomplete text
• Confusing num_words with maxlen