Token-based splitting in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the output of this LangChain TokenTextSplitter?

Given the following code using LangChain's TokenTextSplitter, what will be the number of chunks produced?

LangChain

from langchain.text_splitter import TokenTextSplitter
text = "Hello world! This is a test of token-based splitting."
splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=0)
chunks = splitter.split_text(text)
print(len(chunks))

A 9

B 7

C 10

D 3

💡 Hint

Consider how many tokens the text has and how chunk_size divides them.

The text has 13 tokens approximately. With chunk_size=5 and no overlap, the splitter creates 3 chunks (5 tokens each except the last chunk which may be smaller).

📝 Syntax

intermediate

Which option correctly initializes TokenTextSplitter with overlap?

Which of the following code snippets correctly creates a TokenTextSplitter with chunk size 10 and chunk overlap 3?

A splitter = TokenTextSplitter(10, 3)

B splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=3)

C splitter = TokenTextSplitter(chunk_size=10 chunk_overlap=3)

D splitter = TokenTextSplitter(chunkSize=10, chunkOverlap=3)

💡 Hint

Check the parameter names and syntax for Python function calls.

Option B uses correct parameter names and syntax. Option B uses camelCase which is invalid. Option B misses a comma causing syntax error. Option B passes positional arguments which work but using keywords is preferred for clarity.

🔧 Debug

advanced

Why does this TokenTextSplitter code raise an error?

What error will this code raise and why? from langchain.text_splitter import TokenTextSplitter splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=6) chunks = splitter.split_text("Sample text for splitting.")

LangChain

from langchain.text_splitter import TokenTextSplitter
splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=6)
chunks = splitter.split_text("Sample text for splitting.")

A ValueError because chunk_overlap cannot be greater than chunk_size

B TypeError because chunk_overlap must be a string

C IndexError because chunk_size is too small

D No error, code runs fine

💡 Hint

Think about logical constraints between chunk_size and chunk_overlap.

chunk_overlap must be less than or equal to chunk_size. Here chunk_overlap=6 is greater than chunk_size=5, so a ValueError is raised.

❓ state_output

advanced

What is the content of the first chunk after splitting?

Given this code, what is the exact text of the first chunk produced by TokenTextSplitter? text = "The quick brown fox jumps over the lazy dog." splitter = TokenTextSplitter(chunk_size=4, chunk_overlap=1) chunks = splitter.split_text(text) print(chunks[0])

LangChain

from langchain.text_splitter import TokenTextSplitter
text = "The quick brown fox jumps over the lazy dog."
splitter = TokenTextSplitter(chunk_size=4, chunk_overlap=1)
chunks = splitter.split_text(text)
print(chunks[0])

A "The quick brown fox"

B "The quick brown"

C "The quick brown fox jumps"

D "quick brown fox jumps"

💡 Hint

Remember chunk_size counts tokens, and overlap means next chunk starts 1 token before previous chunk ends.

The first chunk contains 4 tokens: 'The', 'quick', 'brown', 'fox'. So the text is exactly "The quick brown fox".

🧠 Conceptual

expert

How does TokenTextSplitter handle tokenization internally?

Which statement best describes how LangChain's TokenTextSplitter splits text into chunks?

A It splits text into fixed character length chunks without considering tokens.

B It splits text by whitespace and punctuation only, ignoring token semantics.

C It uses a tokenizer from a language model to split text into tokens, then groups tokens into chunks based on chunk_size and chunk_overlap.

D It uses regex patterns to split text into chunks based on sentence boundaries.

💡 Hint

Think about what 'token-based' means in this context.

TokenTextSplitter uses a tokenizer from a language model (like tiktoken) to split text into tokens, then creates chunks of tokens with specified size and overlap.

Practice

1. What is the main purpose of token-based splitting in langchain?

easy

A. To translate text into different languages automatically

B. To combine multiple texts into one large string

C. To encrypt text data for security

D. To break text into smaller token-sized chunks for easier processing

Solution

1. Step 1: Understand token-based splitting

   Token-based splitting divides text into smaller pieces based on tokens, which are units of text like words or subwords.

2. Step 2: Identify the purpose

   This helps manage large texts by breaking them into manageable chunks for language models with token limits.

3. Final Answer:

   To break text into smaller token-sized chunks for easier processing -> Option D

4. Quick Check:

   Token-based splitting = chunking text [OK]

Hint: Token splitting means breaking text into small pieces [OK]

Common Mistakes:

• Confusing splitting with combining texts
• Thinking it encrypts or translates text
• Assuming it changes text meaning

2. Which of the following is the correct way to create a token-based text splitter in langchain?

easy

A. TextSplitter(chunk_size=100, overlap=20)

B. TokenTextSplitter(chunk_size=100, chunk_overlap=20)

C. TokenSplitter(size=100, overlap=20)

D. TokenTextSplitter(size=100, overlap=20)

Solution

1. Step 1: Recall the correct class name and parameters

   The correct class is TokenTextSplitter with parameters chunk_size and chunk_overlap.

2. Step 2: Match options with correct syntax

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) uses TokenTextSplitter with chunk_size and chunk_overlap, which is correct.

3. Final Answer:

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) -> Option B

4. Quick Check:

   Correct class and parameters = TokenTextSplitter(chunk_size=100, chunk_overlap=20) [OK]

Hint: Look for TokenTextSplitter with chunk_size and chunk_overlap [OK]

Common Mistakes:

• Using wrong class names like TokenSplitter
• Using incorrect parameter names like size or overlap
• Mixing parameter names or omitting chunk_ prefix

3. Given the following code snippet, what will be the length of the first chunk if the text has 250 tokens, chunk_size=100, and chunk_overlap=20?

splitter = TokenTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_text(text)

medium

A. 100 tokens

B. 120 tokens

C. 80 tokens

D. 250 tokens

Solution

1. Step 1: Understand chunk size and overlap

   The chunk_size defines the max tokens per chunk, here 100 tokens.

2. Step 2: Determine first chunk length

   The first chunk starts at token 0 and includes 100 tokens; overlap applies to next chunks, not the first.

3. Final Answer:

   100 tokens -> Option A

4. Quick Check:

   First chunk = chunk_size = 100 tokens [OK]

Hint: First chunk length equals chunk_size, overlap affects next chunks [OK]

Common Mistakes:

• Adding overlap to first chunk size
• Assuming first chunk covers entire text
• Confusing overlap as extra tokens in first chunk

4. What is wrong with this code snippet for token-based splitting?

splitter = TokenTextSplitter(chunk_size=50, chunk_overlap=60)
chunks = splitter.split_text(text)

medium

A. split_text method does not exist

B. chunk_size must be a string, not a number

C. chunk_overlap cannot be larger than chunk_size

D. chunk_overlap must be zero

Solution

1. Step 1: Check parameter constraints

   chunk_overlap should not exceed chunk_size because overlap means shared tokens between chunks.

2. Step 2: Identify error in values

   Here, chunk_overlap=60 is greater than chunk_size=50, which is invalid and causes an error.

3. Final Answer:

   chunk_overlap cannot be larger than chunk_size -> Option C

4. Quick Check:

   Overlap ≤ chunk size rule violated [OK]

Hint: Overlap must be less or equal to chunk size [OK]

Common Mistakes:

• Thinking chunk_size must be string
• Assuming split_text method is missing
• Believing overlap must be zero always

5. You have a long document of 1000 tokens and want to split it into chunks of 200 tokens with 50 tokens overlap. How many chunks will you get using TokenTextSplitter(chunk_size=200, chunk_overlap=50)?

hard

A. 7 chunks

B. 6 chunks

C. 8 chunks

D. 5 chunks

Solution

1. Step 1: Calculate effective step size

   Each new chunk starts 200 - 50 = 150 tokens after the previous chunk start.

2. Step 2: Calculate number of chunks

   Number of chunks = ceil((1000 - 50) / 150) = ceil(950 / 150) = ceil(6.33) = 7 chunks.

3. Final Answer:

   7 chunks -> Option A

4. Quick Check:

   Chunks = ceil((total - overlap) / (chunk_size - overlap)) = 7 [OK]

Hint: Use step = chunk_size - overlap, then divide total tokens [OK]

Common Mistakes:

• Ignoring overlap in calculation
• Dividing total tokens by chunk_size directly
• Rounding down instead of up