Token-based splitting in LangChain - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the TokenTextSplitter from langchain.text_splitter.

LangChain

from langchain.text_splitter import [1]

Drag options to blanks, or click blank then click option'

ATokenTextSplitter

BCharacterTextSplitter

CRecursiveCharacterTextSplitter

DSentenceSplitter

Attempts:

3 left

💡 Hint

Common Mistakes

Importing CharacterTextSplitter instead of TokenTextSplitter

Using incorrect class names

Forgetting to import from langchain.text_splitter

The TokenTextSplitter class is used for splitting text based on tokens in Langchain.

2fill in blank

medium

Complete the code to create a TokenTextSplitter instance with chunk size 1000.

LangChain

splitter = TokenTextSplitter(chunk_size=[1])

Drag options to blanks, or click blank then click option'

A2000

B1000

C500

D50

Attempts:

3 left

💡 Hint

Common Mistakes

Using too small chunk size like 50

Using chunk size larger than 2000 without reason

Confusing chunk size with chunk overlap

The chunk_size parameter defines the maximum number of tokens per chunk. Here, 1000 tokens is a common chunk size.

3fill in blank

hard

Fix the error in the code to split text using the TokenTextSplitter instance.

LangChain

chunks = splitter.[1](text)

Drag options to blanks, or click blank then click option'

Atokenize

Bsplit

Csplit_text

Dchunk_text

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'split' instead of 'split_text'

Using 'tokenize' which only tokenizes but does not split

Using non-existent methods like 'chunk_text'

The correct method to split text into chunks is split_text in TokenTextSplitter.

4fill in blank

hard

Fill both blanks to create a TokenTextSplitter with chunk size 500 and chunk overlap 50.

LangChain

splitter = TokenTextSplitter(chunk_size=[1], chunk_overlap=[2])

Drag options to blanks, or click blank then click option'

A500

B100

C50

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Setting overlap larger than chunk size

Using zero overlap when some overlap is needed

Confusing chunk size and overlap values

The chunk_size is set to 500 tokens, and chunk_overlap is set to 50 tokens to allow some overlap between chunks.

5fill in blank

hard

Fill all three blanks to create a TokenTextSplitter with chunk size 800, chunk overlap 100, and a custom encoding 'gpt2'.

LangChain

splitter = TokenTextSplitter(chunk_size=[1], chunk_overlap=[2], encoding_name=[3])

Drag options to blanks, or click blank then click option'

A1000

B100

C'gpt2'

D800

Attempts:

3 left

💡 Hint

Common Mistakes

Using encoding name without quotes

Mixing up chunk size and overlap values

Using unsupported encoding names

This creates a TokenTextSplitter with 800 tokens per chunk, 100 tokens overlap, and uses the 'gpt2' encoding for tokenization.

Practice

1. What is the main purpose of token-based splitting in langchain?

easy

A. To translate text into different languages automatically

B. To combine multiple texts into one large string

C. To encrypt text data for security

D. To break text into smaller token-sized chunks for easier processing

Solution

1. Step 1: Understand token-based splitting

   Token-based splitting divides text into smaller pieces based on tokens, which are units of text like words or subwords.

2. Step 2: Identify the purpose

   This helps manage large texts by breaking them into manageable chunks for language models with token limits.

3. Final Answer:

   To break text into smaller token-sized chunks for easier processing -> Option D

4. Quick Check:

   Token-based splitting = chunking text [OK]

Hint: Token splitting means breaking text into small pieces [OK]

Common Mistakes:

• Confusing splitting with combining texts
• Thinking it encrypts or translates text
• Assuming it changes text meaning

2. Which of the following is the correct way to create a token-based text splitter in langchain?

easy

A. TextSplitter(chunk_size=100, overlap=20)

B. TokenTextSplitter(chunk_size=100, chunk_overlap=20)

C. TokenSplitter(size=100, overlap=20)

D. TokenTextSplitter(size=100, overlap=20)

Solution

1. Step 1: Recall the correct class name and parameters

   The correct class is TokenTextSplitter with parameters chunk_size and chunk_overlap.

2. Step 2: Match options with correct syntax

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) uses TokenTextSplitter with chunk_size and chunk_overlap, which is correct.

3. Final Answer:

   TokenTextSplitter(chunk_size=100, chunk_overlap=20) -> Option B

4. Quick Check:

   Correct class and parameters = TokenTextSplitter(chunk_size=100, chunk_overlap=20) [OK]

Hint: Look for TokenTextSplitter with chunk_size and chunk_overlap [OK]

Common Mistakes:

• Using wrong class names like TokenSplitter
• Using incorrect parameter names like size or overlap
• Mixing parameter names or omitting chunk_ prefix

3. Given the following code snippet, what will be the length of the first chunk if the text has 250 tokens, chunk_size=100, and chunk_overlap=20?

splitter = TokenTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_text(text)

medium

A. 100 tokens

B. 120 tokens

C. 80 tokens

D. 250 tokens

Solution

1. Step 1: Understand chunk size and overlap

   The chunk_size defines the max tokens per chunk, here 100 tokens.

2. Step 2: Determine first chunk length

   The first chunk starts at token 0 and includes 100 tokens; overlap applies to next chunks, not the first.

3. Final Answer:

   100 tokens -> Option A

4. Quick Check:

   First chunk = chunk_size = 100 tokens [OK]

Hint: First chunk length equals chunk_size, overlap affects next chunks [OK]

Common Mistakes:

• Adding overlap to first chunk size
• Assuming first chunk covers entire text
• Confusing overlap as extra tokens in first chunk

4. What is wrong with this code snippet for token-based splitting?

splitter = TokenTextSplitter(chunk_size=50, chunk_overlap=60)
chunks = splitter.split_text(text)

medium

A. split_text method does not exist

B. chunk_size must be a string, not a number

C. chunk_overlap cannot be larger than chunk_size

D. chunk_overlap must be zero

Solution

1. Step 1: Check parameter constraints

   chunk_overlap should not exceed chunk_size because overlap means shared tokens between chunks.

2. Step 2: Identify error in values

   Here, chunk_overlap=60 is greater than chunk_size=50, which is invalid and causes an error.

3. Final Answer:

   chunk_overlap cannot be larger than chunk_size -> Option C

4. Quick Check:

   Overlap ≤ chunk size rule violated [OK]

Hint: Overlap must be less or equal to chunk size [OK]

Common Mistakes:

• Thinking chunk_size must be string
• Assuming split_text method is missing
• Believing overlap must be zero always

5. You have a long document of 1000 tokens and want to split it into chunks of 200 tokens with 50 tokens overlap. How many chunks will you get using TokenTextSplitter(chunk_size=200, chunk_overlap=50)?

hard

A. 7 chunks

B. 6 chunks

C. 8 chunks

D. 5 chunks

Solution

1. Step 1: Calculate effective step size

   Each new chunk starts 200 - 50 = 150 tokens after the previous chunk start.

2. Step 2: Calculate number of chunks

   Number of chunks = ceil((1000 - 50) / 150) = ceil(950 / 150) = ceil(6.33) = 7 chunks.

3. Final Answer:

   7 chunks -> Option A

4. Quick Check:

   Chunks = ceil((total - overlap) / (chunk_size - overlap)) = 7 [OK]

Hint: Use step = chunk_size - overlap, then divide total tokens [OK]

Common Mistakes:

• Ignoring overlap in calculation
• Dividing total tokens by chunk_size directly
• Rounding down instead of up