RecursiveCharacterTextSplitter in LangChain - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of RecursiveCharacterTextSplitter in langchain?

It splits long text into smaller chunks by recursively breaking it down using a list of separators, helping manage text size for processing.

intermediate

How does RecursiveCharacterTextSplitter decide where to split the text?

It tries to split text using the first separator in its list. If chunks are still too large, it recursively uses the next separator until chunks are small enough.

intermediate

Why is recursion useful in RecursiveCharacterTextSplitter?

Recursion allows the splitter to try multiple levels of splitting, starting with bigger breaks and moving to smaller ones, ensuring chunks fit size limits.

advanced

What happens if RecursiveCharacterTextSplitter cannot split text into small enough chunks?

It returns the chunk as is, even if it is larger than the target size, because no smaller separators remain to split further.

beginner

Name two common separators RecursiveCharacterTextSplitter might use.

Common separators include '\n\n' (double newlines) and '\n' (single newline), which help split paragraphs and lines respectively.

What is the first step RecursiveCharacterTextSplitter takes when splitting text?

A Split text by word count

B Split text using the first separator in its list

C Split text randomly

D Split text into single characters

The splitter starts by trying to split text using the first separator it has, like double newlines.

If chunks are still too large after the first split, what does RecursiveCharacterTextSplitter do?

A Uses the next separator in the list to split chunks further

B Merges chunks together

C Stops splitting and returns chunks

D Throws an error

It recursively uses the next separator to split chunks into smaller pieces.

Which of these is NOT a typical separator used by RecursiveCharacterTextSplitter?

A Space ' '

B Single newline '\n'

C Random punctuation '@'

D Double newline '\n\n'

Random punctuation like '@' is not commonly used as a separator in this splitter.

What does RecursiveCharacterTextSplitter return if no smaller separator can split a large chunk?

A The chunk as is, even if large

B An error message

C An empty string

D A truncated chunk

It returns the chunk without further splitting if no smaller separator is available.

Why is RecursiveCharacterTextSplitter useful in text processing?

A It encrypts text for security

B It compresses text to save space

C It translates text into multiple languages

D It splits text into manageable chunks for models or tools with size limits

It helps break down large texts into smaller parts that fit size limits of processing tools.

Explain how RecursiveCharacterTextSplitter uses recursion to split text.

Think about how it tries different separators step by step.

Describe a real-life example where RecursiveCharacterTextSplitter would be helpful.

Imagine you want to feed a big book into a tool that only accepts small parts.

Practice

1. What is the main purpose of the RecursiveCharacterTextSplitter in langchain?

easy

A. To split long text into smaller chunks using multiple separators while keeping context.

B. To merge multiple texts into one large string.

C. To translate text into different languages.

D. To encrypt text for secure storage.

Solution

1. Step 1: Understand the function of RecursiveCharacterTextSplitter

   This tool is designed to break down long text into smaller pieces using various separators like paragraphs, sentences, or characters.

2. Step 2: Recognize the benefit of overlapping chunks

   It keeps some overlap between chunks to maintain context, which helps language models understand the text better.

3. Final Answer:

   To split long text into smaller chunks using multiple separators while keeping context. -> Option A

4. Quick Check:

   Splitting text with context = A [OK]

Hint: Remember: It splits text into chunks with overlap [OK]

Common Mistakes:

• Thinking it merges text instead of splitting
• Confusing it with translation or encryption tools
• Ignoring the overlap feature

2. Which of the following is the correct way to create a RecursiveCharacterTextSplitter with a chunk size of 100 and chunk overlap of 20?

easy

A. RecursiveCharacterTextSplitter(chunkSize=100, chunkOverlap=20)

B. RecursiveCharacterTextSplitter(size=100, overlap=20)

C. RecursiveCharacterTextSplitter(chunk=100, overlap=20)

D. RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)

Solution

1. Step 1: Recall the parameter names for RecursiveCharacterTextSplitter

   The correct parameters are chunk_size and chunk_overlap to set chunk length and overlap.

2. Step 2: Match the correct syntax

   RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) uses the exact parameter names and values correctly.

3. Final Answer:

   RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) -> Option D

4. Quick Check:

   Correct parameter names = D [OK]

Hint: Use exact parameter names: chunk_size and chunk_overlap [OK]

Common Mistakes:

• Using incorrect parameter names like size or chunkSize
• Mixing camelCase and snake_case
• Omitting required parameters

3. Given the following code snippet, what will be the length of the first chunk if the text is "Hello world! This is a test." and chunk_size=12, chunk_overlap=4?

splitter = RecursiveCharacterTextSplitter(chunk_size=12, chunk_overlap=4)
chunks = splitter.split_text("Hello world! This is a test.")
print(len(chunks[0]))

medium

A. 12

B. 16

C. 8

D. 20

Solution

1. Step 1: Understand chunk size and overlap effect

   The chunk size is 12, so each chunk will be at most 12 characters long. Overlap affects next chunks, not the first chunk length.

2. Step 2: Determine first chunk length

   The first chunk will be exactly 12 characters long, as it starts from the beginning and takes up to chunk_size.

3. Final Answer:

   12 -> Option A

4. Quick Check:

   First chunk length = chunk_size = 12 [OK]

Hint: First chunk length equals chunk_size, overlap affects next chunks [OK]

Common Mistakes:

• Adding overlap to first chunk length
• Assuming first chunk is shorter than chunk_size
• Confusing chunk size with total text length

4. What is the error in the following code snippet using RecursiveCharacterTextSplitter?

splitter = RecursiveCharacterTextSplitter(chunk_size=50, chunk_overlap=60)
chunks = splitter.split_text("Some long text here...")

medium

A. Missing import statement for RecursiveCharacterTextSplitter

B. split_text method does not exist

C. chunk_overlap cannot be larger than chunk_size

D. chunk_size must be a string, not an integer

Solution

1. Step 1: Check parameter constraints

   chunk_overlap must be less than or equal to chunk_size to avoid logical errors in splitting.

2. Step 2: Identify the error in given values

   chunk_overlap=60 is greater than chunk_size=50, which is invalid and will cause an error.

3. Final Answer:

   chunk_overlap cannot be larger than chunk_size -> Option C

4. Quick Check:

   Overlap ≤ chunk size rule = B [OK]

Hint: Overlap must never exceed chunk size [OK]

Common Mistakes:

• Ignoring parameter value relationships
• Assuming method names are incorrect without checking
• Confusing data types for parameters

5. You want to split a large document into chunks of 200 characters with 50 characters overlapping, prioritizing splitting first by paragraphs, then sentences, then words. Which setup of RecursiveCharacterTextSplitter correctly achieves this?

hard

A. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[" ", ". ", "\n\n"])

B. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "])

C. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[". ", "\n\n", " "])

D. RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=[" ", "\n\n", ". "])

Solution

1. Step 1: Understand separator priority order

   The splitter tries separators in order; to prioritize paragraphs, then sentences, then words, the list must start with paragraph separator "\n\n", then sentence ". ", then space " ".

2. Step 2: Match the correct separators list

   RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "]) correctly lists separators in the desired priority order.

3. Final Answer:

   RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=50, separators=["\n\n", ". ", " "]) -> Option B

4. Quick Check:

   Separator order matches priority = A [OK]

Hint: List separators from largest to smallest text unit [OK]

Common Mistakes:

• Reversing separator order
• Using wrong separator strings
• Ignoring overlap or chunk size settings