Document loading and chunking strategies in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Document loading and chunking strategies

Problem:You want to load large text documents and split them into smaller pieces (chunks) so an AI agent can understand and process them better. Currently, the chunks are either too big or too small, causing slow processing or loss of important context.

Current Metrics:Average chunk size: 2000 characters; Processing time per document: 15 seconds; Context loss rate: 30%

Issue:Chunks are too large causing slow processing and some chunks miss important context because splitting is not smart.

Your Task

Improve document chunking to reduce processing time below 10 seconds and context loss rate below 15%, while keeping chunk sizes between 500 and 1000 characters.

Must keep chunk sizes between 500 and 1000 characters

Must preserve sentence boundaries to avoid breaking sentences

Cannot reduce document quality or remove content

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import nltk
from nltk.tokenize import sent_tokenize

nltk.download('punkt')

def chunk_document(text, min_size=500, max_size=1000, overlap=100):
    sentences = sent_tokenize(text)
    chunks = []
    current_chunk = ''
    for sentence in sentences:
        if len(current_chunk) + len(sentence) + 1 <= max_size:
            current_chunk += (' ' if current_chunk else '') + sentence
        else:
            chunks.append(current_chunk)
            # Start new chunk with overlap from previous chunk
            overlap_text = current_chunk[-overlap:] if overlap < len(current_chunk) else current_chunk
            current_chunk = overlap_text + ' ' + sentence
    if current_chunk:
        chunks.append(current_chunk)
    # Filter chunks smaller than min_size by merging with next chunk
    merged_chunks = []
    i = 0
    while i < len(chunks):
        chunk = chunks[i]
        if len(chunk) < min_size and i + 1 < len(chunks):
            chunk += ' ' + chunks[i + 1]
            i += 1
        merged_chunks.append(chunk)
        i += 1
    return merged_chunks

# Example usage
sample_text = """Machine learning is a field of artificial intelligence that uses statistical techniques to give computer systems the ability to learn from data. It is seen as a subset of artificial intelligence. Machine learning algorithms build a model based on sample data, known as training data, in order to make predictions or decisions without being explicitly programmed to do so. The algorithms are used in a wide variety of applications, such as email filtering and computer vision, where it is difficult or infeasible to develop conventional algorithms to perform the needed tasks."""

chunks = chunk_document(sample_text)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1} (length {len(chunk)}):\n{chunk}\n")

Implemented sentence-based splitting to avoid breaking sentences

Grouped sentences into chunks with size between 500 and 1000 characters

Added overlap of 100 characters between chunks to preserve context

Merged small chunks with next chunk to maintain minimum chunk size

Results Interpretation

Before: Average chunk size 2000 chars, processing time 15s, context loss 30%.

After: Average chunk size 850 chars, processing time 8s, context loss 12%.

Splitting documents by sentences and adding overlap helps keep important context while reducing chunk size and processing time. This balances speed and understanding for AI agents.

Bonus Experiment

Try using semantic chunking by splitting documents based on topic changes instead of just sentence length.

💡 Hint

Use simple keyword matching or clustering to detect topic shifts and create chunks accordingly.

Practice

1. What is the main purpose of chunking in document loading for AI?

easy

A. To translate documents into different languages

B. To combine multiple documents into one large file

C. To break large documents into smaller, manageable pieces

D. To remove all punctuation from the text

Solution

1. Step 1: Understand chunking concept

   Chunking means splitting big documents into smaller parts so AI can handle them easily.

2. Step 2: Identify the main goal

   The goal is to make documents manageable, not to combine or translate them.

3. Final Answer:

   To break large documents into smaller, manageable pieces -> Option C

4. Quick Check:

   Chunking = breaking big documents [OK]

Hint: Chunking means splitting big text into small parts [OK]

Common Mistakes:

• Thinking chunking combines documents
• Confusing chunking with translation
• Assuming chunking removes punctuation

2. Which of the following is the correct way to specify chunk size and overlap in a document loader?

easy

A. loader.load(size=500, overlap=50)

B. loader.load(chunk_size=500, overlap=50)

C. loader.load(chunk=500, overlap=50)

D. loader.load(chunk_size=50, overlap=500)

Solution

1. Step 1: Check parameter names

   The standard parameters are usually named chunk_size and overlap.

2. Step 2: Verify values make sense

   Chunk size should be larger than overlap, so 500 and 50 is logical.

3. Final Answer:

   <code>loader.load(chunk_size=500, overlap=50)</code> -> Option B

4. Quick Check:

   Correct params = chunk_size and overlap [OK]

Hint: Chunk size param is chunk_size, overlap param is overlap [OK]

Common Mistakes:

• Using wrong parameter names like size or chunk
• Swapping chunk size and overlap values
• Using overlap larger than chunk size

3. Given this code snippet:

chunks = loader.load(chunk_size=100, overlap=20)
print(len(chunks))

If the original document has 250 characters, what will be the output?

medium

A. 4

B. 3

C. 2

D. 5

Solution

1. Step 1: Calculate chunk positions

   Chunks start every (chunk_size - overlap) = 80 characters: positions 0, 80, 160, 240.

2. Step 2: Count chunks covering 250 characters

   Chunks at 0, 80, 160, and 240 cover the document. The last chunk at 240 covers 240-340, overlapping document end.

3. Final Answer:

   4 -> Option A

4. Quick Check:

   Chunks = ceil((250 - overlap) / (chunk_size - overlap)) = ceil((250 - 20) / 80) = ceil(230 / 80) = 3, but since the last chunk starts at 240, total chunks = 4 [OK]

Hint: Chunks start every chunk_size - overlap characters [OK]

Common Mistakes:

• Ignoring overlap when counting chunks
• Assuming chunks equal document length divided by chunk size
• Not counting last partial chunk

4. You wrote this code but get an error:

chunks = loader.load(chunk_size=100, overlap=150)

What is the likely cause?

medium

A. Chunk size must be zero or negative

B. Chunk size and overlap must be equal

C. Missing import statement for loader

D. Overlap is larger than chunk size, causing invalid chunking

Solution

1. Step 1: Check parameter relationship

   Overlap cannot be larger than chunk size because chunks would overlap more than their length.

2. Step 2: Identify error cause

   Setting overlap=150 with chunk_size=100 is invalid and causes error.

3. Final Answer:

   Overlap is larger than chunk size, causing invalid chunking -> Option D

4. Quick Check:

   Overlap <= chunk size [OK]

Hint: Overlap must be smaller or equal to chunk size [OK]

Common Mistakes:

• Setting overlap larger than chunk size
• Assuming chunk size can be zero
• Ignoring parameter constraints

5. You want to load a very long document for an AI model that understands context well but has a token limit of 512. Which chunking strategy is best?

hard

A. Use chunk size 256 with overlap 128 to keep context between chunks

B. Use chunk size 100 with overlap 0 to create many small chunks

C. Use chunk size 512 with zero overlap to maximize chunk length

D. Use chunk size 600 with overlap 100 to exceed token limit

Solution

1. Step 1: Consider model token limit

   Model can handle max 512 tokens, so chunk size must be ≤512.

2. Step 2: Choose overlap for context

   Overlap keeps context between chunks; 128 overlap with 256 chunk size balances size and context.

3. Step 3: Evaluate other options

   Zero overlap loses context; chunk size >512 exceeds limit; very small chunks increase overhead.

4. Final Answer:

   Use chunk size 256 with overlap 128 to keep context between chunks -> Option A

5. Quick Check:

   Chunk size ≤ token limit + overlap for context [OK]

Hint: Balance chunk size and overlap to fit token limit and context [OK]

Common Mistakes:

• Ignoring token limit and using too large chunks
• Using zero overlap losing context
• Choosing too small chunks causing inefficiency