BERT tokenization (WordPiece) in NLP - ML Experiment: Train & Evaluate

Experiment - BERT tokenization (WordPiece)

Problem:You want to tokenize sentences using BERT's WordPiece tokenizer to prepare text data for a BERT model.

Current Metrics:Tokenization is done but the tokens do not match expected WordPiece tokens, causing poor model input quality.

Issue:The current tokenization uses a simple whitespace split instead of WordPiece, leading to incorrect subword tokens and poor model understanding.

Your Task

Replace the simple whitespace tokenizer with BERT's WordPiece tokenizer and verify that tokenization matches expected WordPiece tokens.

Use the Hugging Face transformers library's BertTokenizer.

Do not change the input sentences.

Ensure the output tokens include subword tokens starting with '##' where appropriate.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

from transformers import BertTokenizer

# Initialize the BERT WordPiece tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Sample sentences
sentences = [
    "Playing football is fun.",
    "Unbelievable! This is amazing."
]

# Tokenize using simple whitespace split (incorrect)
simple_tokens = [sentence.split() for sentence in sentences]

# Tokenize using BERT WordPiece tokenizer (correct)
wordpiece_tokens = [tokenizer.tokenize(sentence) for sentence in sentences]

print("Simple tokens:", simple_tokens)
print("WordPiece tokens:", wordpiece_tokens)

Replaced simple whitespace split tokenization with BertTokenizer from Hugging Face.

Used 'bert-base-uncased' pretrained tokenizer to get WordPiece tokens.

Used tokenizer.tokenize() method to get subword tokens with '##' prefix where needed.

Results Interpretation

Before: Tokenization splits sentences by spaces, e.g., 'Playing football' -> ['Playing', 'football'].

After: WordPiece tokenization splits words into subwords, e.g., 'Playing' -> ['play', '##ing'], capturing word structure better.

Using BERT's WordPiece tokenizer breaks words into meaningful subword units, which helps the model understand rare or complex words better and improves overall model input quality.

Bonus Experiment

Try tokenizing sentences with out-of-vocabulary words or misspellings and observe how WordPiece handles them.

💡 Hint

Use sentences with made-up words or typos and see how WordPiece breaks them into known subwords or characters.

Practice

1. What is the main purpose of BERT's WordPiece tokenization?

easy

A. To split words into smaller known pieces for better handling of unknown words

B. To translate text into another language

C. To remove stop words from sentences

D. To convert text into numerical vectors directly

Solution

1. Step 1: Understand WordPiece tokenization

   WordPiece breaks words into smaller parts called tokens, especially for unknown or rare words.

2. Step 2: Identify the purpose of this splitting

   This splitting helps the model recognize parts of words it has seen before, improving understanding.

3. Final Answer:

   To split words into smaller known pieces for better handling of unknown words -> Option A

4. Quick Check:

   WordPiece = splitting unknown words [OK]

Hint: WordPiece breaks unknown words into known parts [OK]

Common Mistakes:

• Thinking WordPiece translates text
• Confusing tokenization with stop word removal
• Assuming WordPiece directly converts text to numbers

2. Which of the following is the correct way to represent the word 'unaffable' using WordPiece tokens?

easy

A. ["un", "##affable"]

B. ["unaffable"]

C. ["un", "aff", "able"]

D. ["un", "##aff", "##able"]

Solution

1. Step 1: Understand WordPiece token format

   WordPiece uses '##' to mark tokens that continue from a previous token.

2. Step 2: Analyze the options

   ["un", "##aff", "##able"] correctly splits 'unaffable' into 'un' + '##aff' + '##able', showing continuation tokens.

3. Final Answer:

   ["un", "##aff", "##able"] -> Option D

4. Quick Check:

   Continuation tokens start with ## [OK]

Hint: Look for '##' prefix on continuation tokens [OK]

Common Mistakes:

• Ignoring '##' prefix for continuation tokens
• Treating whole word as one token always
• Splitting tokens without '##' where needed

3. Given the sentence "Playing football is fun", which is the correct WordPiece tokenization output?

medium

A. ["Play", "##ing", "football", "is", "fun"]

B. ["Playing", "football", "is", "fun"]

C. ["Play", "##ing", "foot", "##ball", "is", "fun"]

D. ["Play", "ing", "foot", "##ball", "is", "fun"]

Solution

1. Step 1: Tokenize 'Playing'

   WordPiece splits 'Playing' into 'Play' and '##ing' because 'Play' is a known root.

2. Step 2: Tokenize 'football'

   It splits 'football' into 'foot' and '##ball' as common subwords.

3. Step 3: Check remaining words

   'is' and 'fun' are common words and remain as single tokens.

4. Final Answer:

   ["Play", "##ing", "foot", "##ball", "is", "fun"] -> Option C

5. Quick Check:

   Known roots + ## continuation tokens [OK]

Hint: Split known roots, add ## for continuations [OK]

Common Mistakes:

• Not splitting compound words like football
• Missing ## prefix on continuation tokens
• Treating all words as single tokens

4. Identify the error in this WordPiece tokenization output for the word 'unhappy': ["un", "happy"]

medium

A. Missing '##' prefix on 'happy' token

B. Incorrect splitting; 'unhappy' should be one token

C. Tokens should be reversed order

D. No error; this is correct tokenization

Solution

1. Step 1: Check token continuation rules

   In WordPiece, tokens after the first must start with '##' to show continuation.

2. Step 2: Analyze given tokens

   'happy' is a continuation of 'un', so it should be '##happy', not 'happy'.

3. Final Answer:

   Missing '##' prefix on 'happy' token -> Option A

4. Quick Check:

   Continuation tokens need '##' prefix [OK]

Hint: Check if continuation tokens start with '##' [OK]

Common Mistakes:

• Forgetting '##' on continuation tokens
• Assuming all tokens are standalone
• Thinking order of tokens matters here

5. You want to tokenize the sentence "The unbreakable bond" using BERT's WordPiece tokenizer. Which tokenization output correctly handles the unknown word 'unbreakable'?

hard

A. ["The", "unbreakable", "bond"]

B. ["The", "un", "##break", "##able", "bond"]

C. ["The", "un", "breakable", "bond"]

D. ["The", "un", "##breakable", "bond"]

Solution

1. Step 1: Understand unknown word handling

   WordPiece breaks unknown words into smaller known subwords with '##' for continuation.

2. Step 2: Analyze 'unbreakable'

   It splits into 'un' + '##break' + '##able' to represent parts seen in vocabulary.

3. Step 3: Check other tokens

   'The' and 'bond' are common words and remain as single tokens.

4. Final Answer:

   ["The", "un", "##break", "##able", "bond"] -> Option B

5. Quick Check:

   Unknown words split into known subwords with ## [OK]

Hint: Split unknown words into known parts with ## prefix [OK]

Common Mistakes:

• Treating unknown words as single tokens
• Missing ## on continuation tokens
• Splitting without ## prefix on continuation