Token-based splitting in LangChain - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the output of this LangChain TokenTextSplitter?

Given the following code using LangChain's TokenTextSplitter, what will be the number of chunks produced?

LangChain

from langchain.text_splitter import TokenTextSplitter
text = "Hello world! This is a test of token-based splitting."
splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=0)
chunks = splitter.split_text(text)
print(len(chunks))

A 9

B 7

C 10

D 3

💡 Hint

Consider how many tokens the text has and how chunk_size divides them.

The text has 13 tokens approximately. With chunk_size=5 and no overlap, the splitter creates 3 chunks (5 tokens each except the last chunk which may be smaller).

📝 Syntax

intermediate

Which option correctly initializes TokenTextSplitter with overlap?

Which of the following code snippets correctly creates a TokenTextSplitter with chunk size 10 and chunk overlap 3?

A splitter = TokenTextSplitter(10, 3)

B splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=3)

C splitter = TokenTextSplitter(chunk_size=10 chunk_overlap=3)

D splitter = TokenTextSplitter(chunkSize=10, chunkOverlap=3)

💡 Hint

Check the parameter names and syntax for Python function calls.

Option B uses correct parameter names and syntax. Option B uses camelCase which is invalid. Option B misses a comma causing syntax error. Option B passes positional arguments which work but using keywords is preferred for clarity.

🔧 Debug

advanced

Why does this TokenTextSplitter code raise an error?

What error will this code raise and why? from langchain.text_splitter import TokenTextSplitter splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=6) chunks = splitter.split_text("Sample text for splitting.")

LangChain

from langchain.text_splitter import TokenTextSplitter
splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=6)
chunks = splitter.split_text("Sample text for splitting.")

A ValueError because chunk_overlap cannot be greater than chunk_size

B TypeError because chunk_overlap must be a string

C IndexError because chunk_size is too small

D No error, code runs fine

💡 Hint

Think about logical constraints between chunk_size and chunk_overlap.

chunk_overlap must be less than or equal to chunk_size. Here chunk_overlap=6 is greater than chunk_size=5, so a ValueError is raised.

❓ state_output

advanced

What is the content of the first chunk after splitting?

Given this code, what is the exact text of the first chunk produced by TokenTextSplitter? text = "The quick brown fox jumps over the lazy dog." splitter = TokenTextSplitter(chunk_size=4, chunk_overlap=1) chunks = splitter.split_text(text) print(chunks[0])

LangChain

from langchain.text_splitter import TokenTextSplitter
text = "The quick brown fox jumps over the lazy dog."
splitter = TokenTextSplitter(chunk_size=4, chunk_overlap=1)
chunks = splitter.split_text(text)
print(chunks[0])

A "The quick brown fox"

B "The quick brown"

C "The quick brown fox jumps"

D "quick brown fox jumps"

💡 Hint

Remember chunk_size counts tokens, and overlap means next chunk starts 1 token before previous chunk ends.

The first chunk contains 4 tokens: 'The', 'quick', 'brown', 'fox'. So the text is exactly "The quick brown fox".

🧠 Conceptual

expert

How does TokenTextSplitter handle tokenization internally?

Which statement best describes how LangChain's TokenTextSplitter splits text into chunks?

A It splits text into fixed character length chunks without considering tokens.

B It splits text by whitespace and punctuation only, ignoring token semantics.

C It uses a tokenizer from a language model to split text into tokens, then groups tokens into chunks based on chunk_size and chunk_overlap.

D It uses regex patterns to split text into chunks based on sentence boundaries.

💡 Hint

Think about what 'token-based' means in this context.

TokenTextSplitter uses a tokenizer from a language model (like tiktoken) to split text into tokens, then creates chunks of tokens with specified size and overlap.