RecursiveCharacterTextSplitter in LangChain - Step-by-Step Execution

Concept Flow - RecursiveCharacterTextSplitter

Input: Large Text

↓

Check if text length <= chunk_size?

Yes→Return text as chunk

No↓

Find separator from list in text

↓

Split text by separator

↓

For each part

↓

Recursively call splitter on part

↓

Collect chunks

↓

Return all chunks

The splitter checks if text is small enough; if not, it finds a separator to split text and recursively splits each part until chunks are small.

Execution Sample

LangChain

splitter = RecursiveCharacterTextSplitter(chunk_size=10, separators=["\n", " ", ""])
chunks = splitter.split_text("Hello world! This is a test.")

Splits the input text into chunks of max 10 characters using newline, space, or character splits recursively.

Execution Table

Step	Text Input	Condition (len <= 10?)	Separator Found	Action	Chunks Collected
1	"Hello world! This is a test."	False (28 > 10)	"\n" not found, " " found	Split by space	[]
2	"Hello"	True (5 <= 10)	N/A	Return ["Hello"]	["Hello"]
3	"world!"	True (6 <= 10)	N/A	Return ["world!"]	["Hello", "world!"]
4	"This"	True (4 <= 10)	N/A	Return ["This"]	["Hello", "world!", "This"]
5	"is"	True (2 <= 10)	N/A	Return ["is"]	["Hello", "world!", "This", "is"]
6	"a"	True (1 <= 10)	N/A	Return ["a"]	["Hello", "world!", "This", "is", "a"]
7	"test."	True (5 <= 10)	N/A	Return ["test."]	["Hello", "world!", "This", "is", "a", "test."]
8	All parts processed	N/A	N/A	Return combined chunks	["Hello", "world!", "This", "is", "a", "test."]

💡 All text parts are smaller than or equal to chunk_size, recursion ends.

Variable Tracker

Variable	Start	After Step 1	After Step 2-7	Final
text	"Hello world! This is a test."	"Hello world! This is a test."	Each part of split text	N/A
chunks	[]	[]	["Hello", "world!", "This", "is", "a", "test."]	["Hello", "world!", "This", "is", "a", "test."]

Key Moments - 3 Insights

Why does the splitter try multiple separators instead of just splitting by space?

The splitter tries separators in order to split text into meaningful chunks. If a separator is not found, it tries the next one. This is shown in Step 1 where newline is not found but space is used.

What happens when a text part is smaller than chunk_size?

When text length is less or equal to chunk_size, the splitter returns it as a chunk without further splitting, as seen in Steps 2-7.

Why is recursion needed here?

Recursion allows splitting large text into smaller parts step-by-step until all chunks fit the size limit, demonstrated by splitting each part recursively.

Visual Quiz - 3 Questions

Look at the execution table, what is the chunk collected after processing the text "is"?

A ["is"]

B ["a"]

C ["This"]

D ["test."]

💡 Hint

Check Step 5 in the execution_table under Chunks Collected.

At which step does the splitter find the first separator to split the text?

A Step 2

B Step 1

C Step 8

D Step 4

💡 Hint

Look at Step 1 in execution_table where separator " " is found.

If chunk_size was increased to 30, how would the execution table change?

A The splitter would split text into more chunks.

B The splitter would not find any separator.

C The splitter would return the whole text as one chunk at Step 1.

D The recursion would go deeper.

💡 Hint

Refer to Condition column in Step 1 and chunk_size comparison.

Concept Snapshot

RecursiveCharacterTextSplitter splits large text into smaller chunks.
It tries separators in order to split text.
If text is small enough, returns it as a chunk.
Otherwise, splits recursively until all chunks fit size.
Useful for processing long texts in manageable pieces.

Full Transcript

The RecursiveCharacterTextSplitter takes a large text and splits it into smaller chunks based on a maximum chunk size. It checks if the text length is less than or equal to the chunk size. If yes, it returns the text as a chunk. If not, it tries to find a separator from a list to split the text. It splits the text by the first found separator and recursively applies the same process to each part. This continues until all chunks are small enough. For example, splitting "Hello world! This is a test." with chunk size 10 and separators newline, space, and empty string results in chunks like "Hello", "world!", "This", "is", "a", "test.". This method helps break down large texts into manageable pieces for further processing.

Practice

1. What is the main purpose of the RecursiveCharacterTextSplitter in langchain?

easy

A. To split long text into smaller chunks using multiple separators while keeping context.

B. To merge multiple texts into one large string.

C. To translate text into different languages.

D. To encrypt text for secure storage.

Solution

1. Step 1: Understand the function of RecursiveCharacterTextSplitter

   This tool is designed to break down long text into smaller pieces using various separators like paragraphs, sentences, or characters.

2. Step 2: Recognize the benefit of overlapping chunks

   It keeps some overlap between chunks to maintain context, which helps language models understand the text better.

3. Final Answer:

   To split long text into smaller chunks using multiple separators while keeping context. -> Option A

4. Quick Check:

   Splitting text with context = A [OK]

Hint: Remember: It splits text into chunks with overlap [OK]

Common Mistakes:

• Thinking it merges text instead of splitting
• Confusing it with translation or encryption tools
• Ignoring the overlap feature

2. Which of the following is the correct way to create a RecursiveCharacterTextSplitter with a chunk size of 100 and chunk overlap of 20?

easy

A. RecursiveCharacterTextSplitter(chunkSize=100, chunkOverlap=20)

B. RecursiveCharacterTextSplitter(size=100, overlap=20)

C. RecursiveCharacterTextSplitter(chunk=100, overlap=20)

D. RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)

Solution

1. Step 1: Recall the parameter names for RecursiveCharacterTextSplitter

   The correct parameters are chunk_size and chunk_overlap to set chunk length and overlap.

2. Step 2: Match the correct syntax

   RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) uses the exact parameter names and values correctly.

3. Final Answer:

   RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) -> Option D

4. Quick Check:

   Correct parameter names = D [OK]

Hint: Use exact parameter names: chunk_size and chunk_overlap [OK]

Common Mistakes:

• Using incorrect parameter names like size or chunkSize
• Mixing camelCase and snake_case
• Omitting required parameters

3. Given the following code snippet, what will be the length of the first chunk if the text is "Hello world! This is a test." and chunk_size=12, chunk_overlap=4?

splitter = RecursiveCharacterTextSplitter(chunk_size=12, chunk_overlap=4)
chunks = splitter.split_text("Hello world! This is a test.")
print(len(chunks[0]))

medium

A. 12

B. 16

C. 8

D. 20

Solution

1. Step 1: Understand chunk size and overlap effect

   The chunk size is 12, so each chunk will be at most 12 characters long. Overlap affects next chunks, not the first chunk length.

2. Step 2: Determine first chunk length

   The first chunk will be exactly 12 characters long, as it starts from the beginning and takes up to chunk_size.

3. Final Answer:

   12 -> Option A

4. Quick Check:

   First chunk length = chunk_size = 12 [OK]

Hint: First chunk length equals chunk_size, overlap affects next chunks [OK]

Common Mistakes:

• Adding overlap to first chunk length
• Assuming first chunk is shorter than chunk_size
• Confusing chunk size with total text length

4. What is the error in the following code snippet using RecursiveCharacterTextSplitter?

splitter = RecursiveCharacterTextSplitter(chunk_size=50, chunk_overlap=60)
chunks = splitter.split_text("Some long text here...")

medium

A. Missing import statement for RecursiveCharacterTextSplitter

B. split_text method does not exist

C. chunk_overlap cannot be larger than chunk_size

D. chunk_size must be a string, not an integer

Solution

1. Step 1: Check parameter constraints

   chunk_overlap must be less than or equal to chunk_size to avoid logical errors in splitting.

2. Step 2: Identify the error in given values

   chunk_overlap=60 is greater than chunk_size=50, which is invalid and will cause an error.

3. Final Answer:

   chunk_overlap cannot be larger than chunk_size -> Option C

4. Quick Check:

   Overlap ≤ chunk size rule = B [OK]

Hint: Overlap must never exceed chunk size [OK]

Common Mistakes:

• Ignoring parameter value relationships
• Assuming method names are incorrect without checking
• Confusing data types for parameters

5. You want to split a large document into chunks of 200 characters with 50 characters overlapping, prioritizing splitting first by paragraphs, then sentences, then words. Which setup of RecursiveCharacterTextSplitter correctly achieves this?

hard

A. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[" ", ". ", "\n\n"])

B. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "])

C. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[". ", "\n\n", " "])

D. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[" ", "\n\n", ". "])

Solution

1. Step 1: Understand separator priority order

   The splitter tries separators in order; to prioritize paragraphs, then sentences, then words, the list must start with paragraph separator "\n\n", then sentence ". ", then space " ".

2. Step 2: Match the correct separators list

   RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "]) correctly lists separators in the desired priority order.

3. Final Answer:

   RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "]) -> Option B

4. Quick Check:

   Separator order matches priority = A [OK]

Hint: List separators from largest to smallest text unit [OK]

Common Mistakes:

• Reversing separator order
• Using wrong separator strings
• Ignoring overlap or chunk size settings