BERT tokenization (WordPiece) in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of BERT tokenization using WordPiece?

To split words into smaller subword units so that rare or unknown words can be represented as combinations of known pieces, improving the model's understanding of language.

beginner

How does WordPiece handle unknown words during tokenization?

It breaks unknown words into smaller known subword units, starting from the beginning of the word and adding pieces until the whole word is covered, allowing the model to understand new words from familiar parts.

beginner

Why does WordPiece add '##' before some tokens?

The '##' symbol marks that the token is a continuation of a previous token and not a standalone word, helping the model know how subwords connect to form full words.

intermediate

Explain the difference between a word and a WordPiece token in BERT tokenization.

A word is a complete unit of language, while a WordPiece token can be a full word or a smaller part of a word. WordPiece tokens allow BERT to handle rare or new words by breaking them into known pieces.

intermediate

What is the advantage of using WordPiece tokenization over simple word-level tokenization?

WordPiece reduces the vocabulary size and handles rare or new words better by splitting them into subwords, which helps the model learn more efficiently and generalize to unseen words.

What does the '##' symbol indicate in WordPiece tokens?

A The token is a suffix or continuation of a previous token

B The token is a prefix of a word

C The token is an unknown word

D The token is a standalone word

The '##' symbol shows that the token continues from the previous token, indicating it is part of the same word.

Why does BERT use WordPiece tokenization instead of splitting only by spaces?

A To increase vocabulary size

B To handle rare and unknown words by breaking them into smaller parts

C To remove punctuation

D To translate words into another language

WordPiece helps BERT handle rare or new words by splitting them into known subword units.

If the word 'unhappiness' is unknown, how might WordPiece tokenize it?

A ['unhappiness']

B ['un', '##happiness']

C ['un', '##happy', '##ness']

D ['unh', '##app', '##iness']

WordPiece breaks words into meaningful subwords like 'un', 'happy', and 'ness' with '##' marking continuations.

What is a key benefit of having a smaller vocabulary with WordPiece?

A Less accurate predictions

B More complex model architecture

C More memory usage

D Faster training and better handling of rare words

A smaller vocabulary reduces model size and helps it learn better by reusing subword units.

Which of these is NOT true about WordPiece tokenization?

A It always treats each word as a single token

B It uses '##' to mark subword continuations

C It splits words into subwords

D It helps handle unknown words

WordPiece often splits words into multiple tokens, especially for rare or unknown words.

Describe how BERT's WordPiece tokenization works and why it is useful.

Think about how breaking words into smaller parts helps the model.

Explain the role of the '##' symbol in WordPiece tokens and give an example.

Consider how subwords connect to form full words.

Practice

1. What is the main purpose of BERT's WordPiece tokenization?

easy

A. To split words into smaller known pieces for better handling of unknown words

B. To translate text into another language

C. To remove stop words from sentences

D. To convert text into numerical vectors directly

Solution

1. Step 1: Understand WordPiece tokenization

   WordPiece breaks words into smaller parts called tokens, especially for unknown or rare words.

2. Step 2: Identify the purpose of this splitting

   This splitting helps the model recognize parts of words it has seen before, improving understanding.

3. Final Answer:

   To split words into smaller known pieces for better handling of unknown words -> Option A

4. Quick Check:

   WordPiece = splitting unknown words [OK]

Hint: WordPiece breaks unknown words into known parts [OK]

Common Mistakes:

• Thinking WordPiece translates text
• Confusing tokenization with stop word removal
• Assuming WordPiece directly converts text to numbers

2. Which of the following is the correct way to represent the word 'unaffable' using WordPiece tokens?

easy

A. ["un", "##affable"]

B. ["unaffable"]

C. ["un", "aff", "able"]

D. ["un", "##aff", "##able"]

Solution

1. Step 1: Understand WordPiece token format

   WordPiece uses '##' to mark tokens that continue from a previous token.

2. Step 2: Analyze the options

   ["un", "##aff", "##able"] correctly splits 'unaffable' into 'un' + '##aff' + '##able', showing continuation tokens.

3. Final Answer:

   ["un", "##aff", "##able"] -> Option D

4. Quick Check:

   Continuation tokens start with ## [OK]

Hint: Look for '##' prefix on continuation tokens [OK]

Common Mistakes:

• Ignoring '##' prefix for continuation tokens
• Treating whole word as one token always
• Splitting tokens without '##' where needed

3. Given the sentence "Playing football is fun", which is the correct WordPiece tokenization output?

medium

A. ["Play", "##ing", "football", "is", "fun"]

B. ["Playing", "football", "is", "fun"]

C. ["Play", "##ing", "foot", "##ball", "is", "fun"]

D. ["Play", "ing", "foot", "##ball", "is", "fun"]

Solution

1. Step 1: Tokenize 'Playing'

   WordPiece splits 'Playing' into 'Play' and '##ing' because 'Play' is a known root.

2. Step 2: Tokenize 'football'

   It splits 'football' into 'foot' and '##ball' as common subwords.

3. Step 3: Check remaining words

   'is' and 'fun' are common words and remain as single tokens.

4. Final Answer:

   ["Play", "##ing", "foot", "##ball", "is", "fun"] -> Option C

5. Quick Check:

   Known roots + ## continuation tokens [OK]

Hint: Split known roots, add ## for continuations [OK]

Common Mistakes:

• Not splitting compound words like football
• Missing ## prefix on continuation tokens
• Treating all words as single tokens

4. Identify the error in this WordPiece tokenization output for the word 'unhappy': ["un", "happy"]

medium

A. Missing '##' prefix on 'happy' token

B. Incorrect splitting; 'unhappy' should be one token

C. Tokens should be reversed order

D. No error; this is correct tokenization

Solution

1. Step 1: Check token continuation rules

   In WordPiece, tokens after the first must start with '##' to show continuation.

2. Step 2: Analyze given tokens

   'happy' is a continuation of 'un', so it should be '##happy', not 'happy'.

3. Final Answer:

   Missing '##' prefix on 'happy' token -> Option A

4. Quick Check:

   Continuation tokens need '##' prefix [OK]

Hint: Check if continuation tokens start with '##' [OK]

Common Mistakes:

• Forgetting '##' on continuation tokens
• Assuming all tokens are standalone
• Thinking order of tokens matters here

5. You want to tokenize the sentence "The unbreakable bond" using BERT's WordPiece tokenizer. Which tokenization output correctly handles the unknown word 'unbreakable'?

hard

A. ["The", "unbreakable", "bond"]

B. ["The", "un", "##break", "##able", "bond"]

C. ["The", "un", "breakable", "bond"]

D. ["The", "un", "##breakable", "bond"]

Solution

1. Step 1: Understand unknown word handling

   WordPiece breaks unknown words into smaller known subwords with '##' for continuation.

2. Step 2: Analyze 'unbreakable'

   It splits into 'un' + '##break' + '##able' to represent parts seen in vocabulary.

3. Step 3: Check other tokens

   'The' and 'bond' are common words and remain as single tokens.

4. Final Answer:

   ["The", "un", "##break", "##able", "bond"] -> Option B

5. Quick Check:

   Unknown words split into known subwords with ## [OK]

Hint: Split unknown words into known parts with ## prefix [OK]

Common Mistakes:

• Treating unknown words as single tokens
• Missing ## on continuation tokens
• Splitting without ## prefix on continuation