Bird
Raised Fist0

Tokenization - BPE, WordPiece, SentencePiece, Why It Matters for LLMs

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Silent Recall Failure
Started after the last model update two days ago.

A sentiment analysis model for social media posts, processing 500k posts per day.

Symptom
Model accuracy remains stable at 0.85, but recall for negative sentiment drops to 0.
📡
Signals
metric
Recall for negative sentiment class
Dropped from 0.78 to 0.00 after the update
log
Tokenization logs showing unusual token frequency
Frequency of '[UNK]' tokens increased by 300%
distribution
Distribution of token lengths in input data
Shifted towards shorter tokens post-update
feature_importance
Importance of 'negative_words' feature
Dropped from 0.25 to 0.05
code
Tokenization script changes
Switched from WordPiece to SentencePiece without retraining
🔬
Hypotheses
Tokenization change caused loss of important tokens for negative sentiment
Supporting Signals
Recall for negative sentiment classFrequency of '[UNK]' tokens increased
Verify: Check if negative sentiment words are being tokenized as '[UNK]'
import collections
from tokenizer import tokenize

negative_words = ['bad', 'terrible', 'awful']
unk_count = collections.Counter(token for word in negative_words for token in tokenize(word))
print(unk_count['[UNK]'])
Model update introduced a bug in handling negative sentiment
Supporting Signals
Recall for negative sentiment classImportance of 'negative_words' feature
Verify: Review model update logs and changes in feature handling
import pandas as pd

feature_importance = pd.read_csv('feature_importance.csv')
print(feature_importance[feature_importance['feature'] == 'negative_words'])
🎯
Root Cause
Tokenization change from WordPiece to SentencePiece without retraining the model.
Why It Happened

The model was not retrained after switching tokenization methods, leading to '[UNK]' tokens for critical words.

🔧
Fix Strategy
ImmediateRollback to the previous model version using WordPiece.
Short-termRetrain the model with the new SentencePiece tokenizer.
Long-termImplement a validation step to ensure tokenization changes are compatible with the model.
🐍
Python Fix
Python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('sentencepiece-model')
model = AutoModel.from_pretrained('sentiment-analysis-model')

# Retrain model
train_model(model, tokenizer)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to small dataset
Why candidates think this: Recall drop could suggest overfitting to positive sentiment.
Why wrong: Accuracy is stable, and '[UNK]' token frequency indicates tokenization issue.
Data pipeline error causing missing data
Why candidates think this: Zero recall might suggest missing negative sentiment data.
Why wrong: Tokenization logs show increased '[UNK]' tokens, not missing data.
📊
Monitoring to Add
  • Alert on sudden changes in '[UNK]' token frequency
  • Monitor recall metrics for each sentiment class
  • Track distribution changes in token lengths
📋
Process Change

Require retraining and validation for any tokenization method changes.

🏗️
Architectural Change

Design models to be robust to tokenization changes by incorporating tokenization as part of the model pipeline.

💡
Key Insight
Tokenization changes can silently disrupt model performance if not validated with retraining.

Practice

(1/5)
1. In the context of model calibration, which metric is most appropriate to evaluate the probability outputs of a classifier?
easy
A. Accuracy
B. Precision
C. Log Loss
D. F1 Score

Solution

  1. Step 1: Identify the purpose of calibration.

    Calibration aims to ensure that predicted probabilities reflect true likelihoods.

  2. Step 2: Evaluate each metric.

    Accuracy, precision, and F1 score do not consider probability estimates, only classification decisions. Log Loss penalizes incorrect probability estimates, making it suitable for calibration.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Log Loss evaluates probability accuracy. [OK]
Hint: Log Loss evaluates probability accuracy. [OK]
Common Mistakes:
2. 
import numpy as np
from sklearn.decomposition import PCA

X = np.array([[1, 2], [3, 4], [5, 6]])
pca = PCA(n_components=1)
X_transformed = pca.fit_transform(X)
print(X_transformed)
What is the shape of X_transformed?
easy
A. (3, 2)
B. (1, 3)
C. (2, 1)
D. (3, 1)

Solution

  1. Step 1: Understand PCA output shape

    PCA reduces the dimensionality of the data to the specified number of components.

  2. Step 2: Calculate the expected shape

    With 3 samples and 1 component, the output shape is (3, 1).

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    PCA output shape: (n_samples, n_components) [OK]
Hint: PCA output shape: (n_samples, n_components) [OK]
Common Mistakes:
3. Which regularization technique is most suitable for feature selection in a linear regression model?
easy
A. L1 Lasso
B. Elastic Net
C. L2 Ridge
D. Dropout

Solution

  1. Step 1: Identify the purpose of feature selection.

    L1 Lasso regularization can shrink some coefficients to zero, effectively selecting features.

  2. Step 2: Eliminate wrong options.

    L2 Ridge does not shrink coefficients to zero. Elastic Net combines L1 and L2 but is not as direct for feature selection. Dropout is not used in linear regression.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    L1 Lasso zeros coefficients for selection. [OK]
Hint: L1 Lasso zeros coefficients for selection. [OK]
Common Mistakes:
4. You implemented stratified sampling but notice that the sample still has class imbalance. What is the most likely cause?
medium
A. Data leakage from training set.
B. Using random sampling instead.
C. Incorrectly specified class proportions.
D. Insufficient sample size.

Solution

  1. Step 1: Identify the stratified sampling requirement.

    Stratified sampling requires correct class proportion specification.

  2. Step 2: Eliminate unrelated issues.

    Random Sampling (B) would not be used if stratified was intended. Data leakage (C) affects model performance, not sampling. Sample size (D) affects variability, not imbalance directly.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check class proportions in stratified sampling [[check]]
Hint: Check class proportions in stratified sampling [[check]]
Common Mistakes:
5. You need to deploy an object detection model on a mobile device with limited computational resources. Which approach would you take?
hard
A. Use a lightweight version of YOLO, like Tiny YOLO
B. Use a full YOLO model for its speed
C. Use a full R-CNN model for its accuracy
D. Use a lightweight version of R-CNN

Solution

  1. Step 1: Consider resource constraints

    Mobile devices require models that are both fast and lightweight.

  2. Step 2: Match model to constraints

    Tiny YOLO is designed for environments with limited resources, balancing speed and size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Mobile = Tiny YOLO [OK]
Hint: Mobile = Tiny YOLO [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard