Tokenization and vocabulary in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Tokenization and vocabulary

This pipeline shows how raw text is changed into tokens and then mapped to a vocabulary for a language model to understand and use.

Data Flow - 3 Stages

1Raw Text Input

1 sentence (string)→Input sentence to be processed→1 sentence (string)

"Hello, how are you?"

↓

2Tokenization

1 sentence (string)→Split sentence into smaller pieces called tokens→List of tokens (e.g., 6 tokens)

["Hello", ",", "how", "are", "you", "?"]

↓

3Vocabulary Mapping

List of tokens (6 tokens)→Convert tokens to numbers using a vocabulary dictionary→List of token IDs (6 integers)

[1543, 12, 78, 45, 89, 7]

Training Trace - Epoch by Epoch

Loss
2.3 |*****
1.85|****
1.4 |***
1.1 |**
0.85|*

Epoch	Loss ↓	Accuracy ↑	Observation
1	2.30	0.15	Model starts with high loss and low accuracy as it learns token patterns.
2	1.85	0.35	Loss decreases and accuracy improves as vocabulary mapping becomes clearer.
3	1.40	0.55	Model better understands token sequences, improving predictions.
4	1.10	0.70	Vocabulary usage is more accurate, loss continues to drop.
5	0.85	0.80	Model converges well on token patterns and vocabulary.

Prediction Trace - 3 Layers

Layer 1: Input Sentence

Layer 2: Tokenization

Layer 3: Vocabulary Mapping

Model Quiz - 3 Questions

What does tokenization do in this pipeline?

A Trains the model to predict next words

B Converts tokens into numbers

C Splits text into smaller pieces called tokens

D Calculates accuracy of the model

Key Insight

Tokenization breaks text into manageable pieces, and vocabulary mapping turns these pieces into numbers. This process helps the model learn language patterns effectively, shown by decreasing loss and increasing accuracy during training.