Bird
Raised Fist0

Tokenization - BPE, WordPiece, SentencePiece, Why It Matters for LLMs

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
BPE vs WordPiece

For a multilingual LLM with a focus on low-resource languages, would you choose BPE or WordPiece for tokenization? Justify your choice.

📊
Comparison
DimensionBPEWordPiece
Vocabulary SizeHighMedium
Handling MorphologyLowHigh
Computational EfficiencyHighMedium
Multilingual SupportMediumHigh
Implementation ComplexityLowMedium
✅
Verdict
Default to BPE for large, morphologically simple datasets with limited resources. Switch to WordPiece when dealing with complex languages or multilingual datasets with sufficient computational resources.
💡
Key Insight
Tokenization choice hinges on balancing vocabulary size, language complexity, and computational resources.
⚠️
Interview Trap
❌ Wrong: Always use BPE because it handles large vocabularies efficiently.
This ignores the complexity of the language and the need for finer granularity in multilingual settings.
✅ Consider the language complexity and resource availability; WordPiece may be better for complex or multilingual scenarios.
BPE
"Efficiently handles large vocabularies by merging frequent pairs of characters or subwords."
Weaknesses
  • May produce suboptimal tokenization for rare words due to frequency-based merges.
  • Struggles with languages that have complex morphology.
  • Can lead to large vocabulary sizes if not carefully managed.
Ideal When
  • When n > 100K and the language is morphologically simple.
  • When you have a large corpus with frequent word repetitions.
  • When computational resources are limited and you need a straightforward implementation.
OpenAI's GPT-2 uses BPE for its ability to handle large vocabularies efficiently.
WordPiece
"Balances vocabulary size and coverage by using a probabilistic model to select subword units."
Weaknesses
  • Requires more computational resources due to its probabilistic nature.
  • Can be less efficient for very large datasets due to iterative training.
  • May not capture rare word nuances as effectively as SentencePiece.
Ideal When
  • When n < 100K and you need to handle multiple languages.
  • When you have access to significant computational resources.
  • When the language has complex morphology and you need finer granularity.
Google's BERT uses WordPiece to effectively manage vocabulary size while maintaining language coverage.
🐍
BPE - Example
▼ Show
Python
import re
from collections import defaultdict

def bpe_encode(text, vocab):
    words = text.split()
    for _ in range(10):
        pairs = defaultdict(int)
        for word in words:
            symbols = list(word)
            for i in range(len(symbols) - 1):
                pairs[(symbols[i], symbols[i + 1])] += 1
        best_pair = max(pairs, key=pairs.get)
        vocab.append(''.join(best_pair))
        words = [re.sub(' '.join(best_pair), ''.join(best_pair), word) for word in words]
    return words
🐍
WordPiece - Example
▼ Show
Python
from collections import Counter

def wordpiece_encode(text, vocab):
    words = text.split()
    for _ in range(10):
        subwords = Counter()
        for word in words:
            for i in range(1, len(word) + 1):
                subwords[word[:i]] += 1
        best_subword = max(subwords, key=subwords.get)
        vocab.append(best_subword)
        words = [word.replace(best_subword, '') if best_subword in word else word for word in words]
    return words
✅
Use BPE When
  • •When n > 100K and the language is morphologically simple.
  • •When you need a quick setup with limited computational resources.
  • •When the dataset has frequent word repetitions.
✅
Use WordPiece When
  • •When n < 100K and you need to handle multiple languages.
  • •When you have access to significant computational resources.
  • •When the language has complex morphology.
↳
Nuances
  • BPE can struggle with languages that have complex morphology, even with large datasets.
  • WordPiece's probabilistic model can be computationally expensive, making it less ideal for very large datasets without sufficient resources.
  • In multilingual settings, WordPiece often provides better coverage due to its finer granularity.

Practice

(1/5)
1. Given the following Python code snippet, what will be the output if the model is trained with biased data?
from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 0, 0]

accuracy = accuracy_score(y_true, y_pred)
print(accuracy)
easy
A. 0.4
B. 0.8
C. 0.6
D. 0.2

Solution

  1. Step 1: Calculate the accuracy.

    Accuracy is the ratio of correctly predicted instances to the total instances. Here, 3 out of 5 predictions are correct.

  2. Step 2: Compute the accuracy.

    Correct predictions: [0, 1, 1] -> 0.6

  3. Final Answer:

    0.6 -> 0.6

  4. Quick Check:

    Count correct predictions, divide by total. [OK]
Hint: Count correct predictions, divide by total. [OK]
Common Mistakes:
2. Given the following PyTorch code snippet, what will be the output shape of the model's output after fine-tuning?
import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.features = nn.Sequential(
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 128)
        )
        self.classifier = nn.Linear(128, 10)

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x

model = SimpleModel()
input_data = torch.randn(32, 512)
output = model(input_data)
print(output.shape)
easy
A. torch.Size([32, 512])
B. torch.Size([32, 128])
C. torch.Size([32, 10])
D. torch.Size([32, 256])

Solution

  1. Step 1: Trace the forward pass.

    The input shape is (32, 512). After passing through the features, it becomes (32, 128). The classifier then outputs (32, 10).

  2. Step 2: Eliminate wrong options.

    torch.Size([32, 512]) is incorrect because the output is not the same as the input. torch.Size([32, 256]) and C are intermediate shapes, not the final output.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Final layer determines output shape. [check]
Hint: Final layer determines output shape. [check]
Common Mistakes:
3. A logistic regression model is giving unexpectedly poor predictions. What is a likely cause?
medium
A. The features are not scaled properly, affecting convergence.
B. The model is underfitting because of insufficient data.
C. The model is overfitting due to too many features.
D. The learning rate is too high, causing divergence.

Solution

  1. Step 1: Identify common issues in logistic regression.

    Feature scaling is crucial for convergence in logistic regression, especially with gradient-based optimization.

  2. Step 2: Eliminate wrong options.

    The model is overfitting due to too many features. is less likely as logistic regression is less prone to overfitting. The model is underfitting because of insufficient data. is incorrect because underfitting is more about model complexity. The learning rate is too high, causing divergence. is incorrect as logistic regression typically uses small learning rates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Check feature scaling for convergence issues [OK]
Hint: Check feature scaling for convergence issues [OK]
Common Mistakes:
4. You are tasked with designing a monitoring system for an ML model in production. Which approach ensures robust detection of both data drift and performance decay?
hard
A. Implement a feedback loop using user satisfaction surveys.
B. Use a combination of statistical tests and performance metrics.
C. Rely solely on periodic model retraining.
D. Monitor only the model's prediction confidence scores.

Solution

  1. Step 1: Identify comprehensive monitoring strategies

    Combining statistical tests with performance metrics covers both drift and decay.

  2. Step 2: Eliminate wrong options

    Implement a feedback loop using user satisfaction surveys. is subjective and indirect. Rely solely on periodic model retraining. is reactive, not proactive. Monitor only the model's prediction confidence scores. is too narrow in scope.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Combine stats and metrics for full coverage [OK]
Hint: Combine stats and metrics for full coverage [OK]
Common Mistakes:
5. You are tasked with developing a system that can autonomously drive a car in a simulated environment. Which learning approach should you prioritize and why?
hard
A. Supervised learning, because it can learn from labeled driving data.
B. Unsupervised learning, because it can discover patterns in driving data.
C. Reinforcement learning, because it can learn optimal driving strategies through trial and error.
D. Supervised learning, because it can classify different driving scenarios.

Solution

  1. Step 1: Identify the task requirements.

    The task requires learning optimal strategies in a dynamic environment.

  2. Step 2: Eliminate wrong options.

    Supervised learning, because it can learn from labeled driving data. is incorrect because labeled data alone doesn't adapt to new scenarios. Unsupervised learning, because it can discover patterns in driving data. is incorrect because pattern discovery doesn't imply strategy learning. Supervised learning, because it can classify different driving scenarios. is incorrect because classification doesn't optimize actions.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Dynamic strategy learning -> Reinforcement learning [OK]
Hint: Dynamic strategy learning -> Reinforcement learning [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard