Token-based splitting in LangChain - Step-by-Step Execution

Concept Flow - Token-based splitting

Input Text

↓

Initialize Tokenizer

↓

Split Text into Tokens

↓

Group Tokens into Chunks

↓

Return List of Text Chunks

The text is first tokenized, then tokens are grouped into chunks, and finally the chunks are returned as split text.

Execution Sample

LangChain

from langchain.text_splitter import TokenTextSplitter

text = "Hello world! This is a test of token splitting."
splitter = TokenTextSplitter(chunk_size=5)
chunks = splitter.split_text(text)
print(chunks)

This code splits the input text into chunks of 5 tokens each using Langchain's TokenTextSplitter.

Execution Table

Step	Action	Tokens Processed	Current Chunk Tokens	Chunks Formed
1	Initialize tokenizer and input text	0
2	Tokenize text	0
3	Add token 'Hello' to current chunk	1	Hello
4	Add token 'world' to current chunk	2	Hello, world
5	Add token '!' to current chunk	3	Hello, world, !
6	Add token 'This' to current chunk	4	Hello, world, !, This
7	Add token 'is' to current chunk	5	Hello, world, !, This, is
8	Current chunk reached chunk_size=5, save chunk	5		Hello world! This is
9	Add token 'a' to new chunk	6	a	Hello world! This is
10	Add token 'test' to current chunk	7	a, test	Hello world! This is
11	Add token 'of' to current chunk	8	a, test, of	Hello world! This is
12	Add token 'token' to current chunk	9	a, test, of, token	Hello world! This is
13	Add token 'splitting' to current chunk	10	a, test, of, token, splitting	Hello world! This is
14	Current chunk reached chunk_size=5, save chunk	10		Hello world! This is, a test of token splitting
15	Add token '.' to new chunk	11	.	Hello world! This is, a test of token splitting
16	End of tokens, save last chunk	11		Hello world! This is, a test of token splitting, .
17	Return all chunks	11		Hello world! This is, a test of token splitting, .

💡 All tokens processed and chunks returned.

Variable Tracker

Variable	Start	After Step 8	After Step 14	After Step 16	Final
tokens_processed	0	5	10	11	11
current_chunk_tokens
chunks_formed		Hello world! This is	Hello world! This is, a test of token splitting	Hello world! This is, a test of token splitting, .	Hello world! This is, a test of token splitting, .

Key Moments - 2 Insights

Why does the current chunk reset after reaching chunk_size?

Because once the chunk reaches the specified token limit (chunk_size=5), it is saved to the chunks list and the current chunk is cleared to start collecting the next group of tokens. See execution_table rows 7-8 and 13-14.

What happens to leftover tokens that don't fill a full chunk?

Leftover tokens form a smaller chunk at the end. In this example, the last token '.' forms its own chunk as shown in execution_table rows 15-16.

Visual Quiz - 3 Questions

Look at the execution table at step 7. How many tokens are in the current chunk?

A 4

B 3

C 5

D 6

💡 Hint

Check the 'Current Chunk Tokens' column at step 7 in the execution_table.

At which step does the first chunk get saved to the chunks list?

A Step 8

B Step 10

C Step 5

D Step 14

💡 Hint

Look for when 'Chunks Formed' first contains a chunk in the execution_table.

If chunk_size was changed to 3, how would the number of chunks change?

A Fewer chunks would be created

B More chunks would be created

C Number of chunks stays the same

D No chunks would be created

💡 Hint

Smaller chunk_size means smaller groups, so more chunks overall. Refer to chunk_size effect in concept_flow.

Concept Snapshot

Token-based splitting:
- Input text is split into tokens.
- Tokens are grouped into chunks of fixed size.
- Each chunk is joined back to text.
- Leftover tokens form a smaller last chunk.
- Useful for processing text in manageable pieces.

Full Transcript

Token-based splitting takes a long text and breaks it into smaller parts based on tokens. First, the text is split into tokens using a tokenizer. Then tokens are collected into groups called chunks, each with a fixed number of tokens. When a chunk reaches the set size, it is saved and a new chunk starts. If tokens remain at the end that don't fill a full chunk, they form a smaller chunk. This method helps handle large texts by splitting them into smaller, manageable pieces for processing.

Practice

1. What is the main purpose of token-based splitting in langchain?

easy

A. To translate text into different languages automatically

B. To combine multiple texts into one large string

C. To encrypt text data for security

D. To break text into smaller token-sized chunks for easier processing

Solution

1. Step 1: Understand token-based splitting

   Token-based splitting divides text into smaller pieces based on tokens, which are units of text like words or subwords.

2. Step 2: Identify the purpose

   This helps manage large texts by breaking them into manageable chunks for language models with token limits.

3. Final Answer:

   To break text into smaller token-sized chunks for easier processing -> Option D

4. Quick Check:

   Token-based splitting = chunking text [OK]

Hint: Token splitting means breaking text into small pieces [OK]

Common Mistakes:

• Confusing splitting with combining texts
• Thinking it encrypts or translates text
• Assuming it changes text meaning

2. Which of the following is the correct way to create a token-based text splitter in langchain?

easy

A. TextSplitter(chunk_size=100, overlap=20)

B. TokenTextSplitter(chunk_size=100, chunk_overlap=20)

C. TokenSplitter(size=100, overlap=20)

D. TokenTextSplitter(size=100, overlap=20)

Solution

1. Step 1: Recall the correct class name and parameters

   The correct class is TokenTextSplitter with parameters chunk_size and chunk_overlap.

2. Step 2: Match options with correct syntax

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) uses TokenTextSplitter with chunk_size and chunk_overlap, which is correct.

3. Final Answer:

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) -> Option B

4. Quick Check:

   Correct class and parameters = TokenTextSplitter(chunk_size=100, chunk_overlap=20) [OK]

Hint: Look for TokenTextSplitter with chunk_size and chunk_overlap [OK]

Common Mistakes:

• Using wrong class names like TokenSplitter
• Using incorrect parameter names like size or overlap
• Mixing parameter names or omitting chunk_ prefix

3. Given the following code snippet, what will be the length of the first chunk if the text has 250 tokens, chunk_size=100, and chunk_overlap=20?

splitter = TokenTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_text(text)

medium

A. 100 tokens

B. 120 tokens

C. 80 tokens

D. 250 tokens

Solution

1. Step 1: Understand chunk size and overlap

   The chunk_size defines the max tokens per chunk, here 100 tokens.

2. Step 2: Determine first chunk length

   The first chunk starts at token 0 and includes 100 tokens; overlap applies to next chunks, not the first.

3. Final Answer:

   100 tokens -> Option A

4. Quick Check:

   First chunk = chunk_size = 100 tokens [OK]

Hint: First chunk length equals chunk_size, overlap affects next chunks [OK]

Common Mistakes:

• Adding overlap to first chunk size
• Assuming first chunk covers entire text
• Confusing overlap as extra tokens in first chunk

4. What is wrong with this code snippet for token-based splitting?

splitter = TokenTextSplitter(chunk_size=50, chunk_overlap=60)
chunks = splitter.split_text(text)

medium

A. split_text method does not exist

B. chunk_size must be a string, not a number

C. chunk_overlap cannot be larger than chunk_size

D. chunk_overlap must be zero

Solution

1. Step 1: Check parameter constraints

   chunk_overlap should not exceed chunk_size because overlap means shared tokens between chunks.

2. Step 2: Identify error in values

   Here, chunk_overlap=60 is greater than chunk_size=50, which is invalid and causes an error.

3. Final Answer:

   chunk_overlap cannot be larger than chunk_size -> Option C

4. Quick Check:

   Overlap ≤ chunk size rule violated [OK]

Hint: Overlap must be less or equal to chunk size [OK]

Common Mistakes:

• Thinking chunk_size must be string
• Assuming split_text method is missing
• Believing overlap must be zero always

5. You have a long document of 1000 tokens and want to split it into chunks of 200 tokens with 50 tokens overlap. How many chunks will you get using TokenTextSplitter(chunk_size=200, chunk_overlap=50)?

hard

A. 7 chunks

B. 6 chunks

C. 8 chunks

D. 5 chunks

Solution

1. Step 1: Calculate effective step size

   Each new chunk starts 200 - 50 = 150 tokens after the previous chunk start.

2. Step 2: Calculate number of chunks

   Number of chunks = ceil((1000 - 50) / 150) = ceil(950 / 150) = ceil(6.33) = 7 chunks.

3. Final Answer:

   7 chunks -> Option A

4. Quick Check:

   Chunks = ceil((total - overlap) / (chunk_size - overlap)) = 7 [OK]

Hint: Use step = chunk_size - overlap, then divide total tokens [OK]

Common Mistakes:

• Ignoring overlap in calculation
• Dividing total tokens by chunk_size directly
• Rounding down instead of up