Token-based splitting in LangChain - Step-by-Step Execution

Concept Flow - Token-based splitting

Input Text

↓

Initialize Tokenizer

↓

Split Text into Tokens

↓

Group Tokens into Chunks

↓

Return List of Text Chunks

The text is first tokenized, then tokens are grouped into chunks, and finally the chunks are returned as split text.

Execution Sample

LangChain

from langchain.text_splitter import TokenTextSplitter

text = "Hello world! This is a test of token splitting."
splitter = TokenTextSplitter(chunk_size=5)
chunks = splitter.split_text(text)
print(chunks)

This code splits the input text into chunks of 5 tokens each using Langchain's TokenTextSplitter.

Execution Table

Step	Action	Tokens Processed	Current Chunk Tokens	Chunks Formed
1	Initialize tokenizer and input text	0
2	Tokenize text	0
3	Add token 'Hello' to current chunk	1	Hello
4	Add token 'world' to current chunk	2	Hello, world
5	Add token '!' to current chunk	3	Hello, world, !
6	Add token 'This' to current chunk	4	Hello, world, !, This
7	Add token 'is' to current chunk	5	Hello, world, !, This, is
8	Current chunk reached chunk_size=5, save chunk	5		Hello world! This is
9	Add token 'a' to new chunk	6	a	Hello world! This is
10	Add token 'test' to current chunk	7	a, test	Hello world! This is
11	Add token 'of' to current chunk	8	a, test, of	Hello world! This is
12	Add token 'token' to current chunk	9	a, test, of, token	Hello world! This is
13	Add token 'splitting' to current chunk	10	a, test, of, token, splitting	Hello world! This is
14	Current chunk reached chunk_size=5, save chunk	10		Hello world! This is, a test of token splitting
15	Add token '.' to new chunk	11	.	Hello world! This is, a test of token splitting
16	End of tokens, save last chunk	11		Hello world! This is, a test of token splitting, .
17	Return all chunks	11		Hello world! This is, a test of token splitting, .

💡 All tokens processed and chunks returned.

Variable Tracker

Variable	Start	After Step 8	After Step 14	After Step 16	Final
tokens_processed	0	5	10	11	11
current_chunk_tokens
chunks_formed		Hello world! This is	Hello world! This is, a test of token splitting	Hello world! This is, a test of token splitting, .	Hello world! This is, a test of token splitting, .

Key Moments - 2 Insights

Why does the current chunk reset after reaching chunk_size?

Because once the chunk reaches the specified token limit (chunk_size=5), it is saved to the chunks list and the current chunk is cleared to start collecting the next group of tokens. See execution_table rows 7-8 and 13-14.

What happens to leftover tokens that don't fill a full chunk?

Leftover tokens form a smaller chunk at the end. In this example, the last token '.' forms its own chunk as shown in execution_table rows 15-16.

Visual Quiz - 3 Questions

Look at the execution table at step 7. How many tokens are in the current chunk?

A 4

B 3

C 5

D 6

💡 Hint

Check the 'Current Chunk Tokens' column at step 7 in the execution_table.

At which step does the first chunk get saved to the chunks list?

A Step 8

B Step 10

C Step 5

D Step 14

💡 Hint

Look for when 'Chunks Formed' first contains a chunk in the execution_table.

If chunk_size was changed to 3, how would the number of chunks change?

A Fewer chunks would be created

B More chunks would be created

C Number of chunks stays the same

D No chunks would be created

💡 Hint

Smaller chunk_size means smaller groups, so more chunks overall. Refer to chunk_size effect in concept_flow.

Concept Snapshot

Token-based splitting:
- Input text is split into tokens.
- Tokens are grouped into chunks of fixed size.
- Each chunk is joined back to text.
- Leftover tokens form a smaller last chunk.
- Useful for processing text in manageable pieces.

Full Transcript

Token-based splitting takes a long text and breaks it into smaller parts based on tokens. First, the text is split into tokens using a tokenizer. Then tokens are collected into groups called chunks, each with a fixed number of tokens. When a chunk reaches the set size, it is saved and a new chunk starts. If tokens remain at the end that don't fill a full chunk, they form a smaller chunk. This method helps handle large texts by splitting them into smaller, manageable pieces for processing.