Text chunking strategies in Prompt Engineering / GenAI - Full Explanation

Introduction

When working with large amounts of text, it can be hard to process or understand everything at once. Breaking text into smaller, manageable pieces helps computers and people handle information more easily and accurately.

Explanation

Fixed-size chunking

This method splits text into equal-sized pieces, like cutting a long rope into equal segments. It does not consider the meaning or structure of the text, just the length. This makes it simple but can cut sentences or ideas in awkward places.

Fixed-size chunking divides text by length without considering meaning or sentence boundaries.

Sentence-based chunking

Here, text is divided by sentences. Each chunk contains one or more complete sentences, preserving meaning better than fixed-size chunks. This helps keep ideas intact but can result in chunks of varying sizes.

Sentence-based chunking keeps sentences whole to preserve meaning.

Semantic chunking

This strategy breaks text based on meaning or topics. It groups related sentences or paragraphs together, so each chunk covers a specific idea. This approach helps computers understand context but requires more complex analysis.

Semantic chunking groups text by meaning to keep related ideas together.

Overlap chunking

Overlap chunking creates chunks that share some text with neighboring chunks. This overlap helps maintain context between chunks, reducing the chance of losing important connections when processing each piece separately.

Overlap chunking shares text between chunks to preserve context.

Real World Analogy

Imagine you have a long storybook to share with friends. You can cut it into equal pages, split it by chapters, group parts by themes, or share some sentences twice between friends to keep the story connected.

Fixed-size chunking → Cutting the storybook into equal pages without caring about sentences or chapters

Sentence-based chunking → Splitting the storybook by chapters or paragraphs so each friend gets a complete part

Semantic chunking → Grouping story parts by themes like adventure or mystery to keep related ideas together

Overlap chunking → Sharing some sentences twice between friends so they don’t miss important connections

Diagram

Diagram

┌───────────────┐
│   Full Text   │
└──────┬────────┘
       │
       ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│ Fixed-size    │   │ Sentence-based │   │ Semantic      │   │ Overlap       │
│ chunks        │   │ chunks        │   │ chunks        │   │ chunks        │
│ [equal parts] │   │ [by sentences]│   │ [by meaning]  │   │ [shared text] │
└───────────────┘   └───────────────┘   └───────────────┘   └───────────────┘

This diagram shows the full text being divided into four types of chunks: fixed-size, sentence-based, semantic, and overlap.

Key Facts

Fixed-size chunking → Splits text into equal-length pieces without considering meaning.

Sentence-based chunking → Divides text by complete sentences to keep ideas intact.

Semantic chunking → Groups text by meaning or topic to preserve context.

Overlap chunking → Creates chunks that share some text to maintain connections.

Common Confusions

Thinking fixed-size chunks always keep sentences whole.

Thinking fixed-size chunks always keep sentences whole. Fixed-size chunking cuts text purely by length, so sentences can be split across chunks.

Believing semantic chunking is simple to implement.

Believing semantic chunking is simple to implement. Semantic chunking requires understanding text meaning, which needs advanced analysis and is more complex.

Summary

Breaking text into chunks helps manage and understand large amounts of information.

Different chunking strategies balance simplicity and preserving meaning in various ways.

Choosing the right chunking method depends on the goal and how much context needs to be kept.

Practice

1. What is the main purpose of text chunking in AI models?

easy

A. To generate new text from scratch

B. To split long text into smaller, manageable pieces

C. To remove stop words from text

D. To translate text into different languages

Solution

1. Step 1: Understand the concept of text chunking

   Text chunking means breaking a long text into smaller parts so it is easier to handle.

2. Step 2: Identify the main goal in AI context

   This helps AI models process and understand large texts better by working on smaller pieces.

3. Final Answer:

   To split long text into smaller, manageable pieces -> Option B

4. Quick Check:

   Text chunking = splitting text [OK]

Hint: Chunking means breaking text into smaller parts [OK]

Common Mistakes:

• Confusing chunking with translation
• Thinking chunking removes words
• Believing chunking generates new text

2. Which of the following is a correct way to create overlapping text chunks in Python?

easy

A. chunks = [text[i:i+chunk_size] for i in range(0, len(text), overlap)]

B. chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

C. chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)]

D. chunks = [text[i:i+chunk_size] for i in range(overlap, len(text), chunk_size)]

Solution

1. Step 1: Understand overlapping chunk logic

   To create overlapping chunks, the step size must be smaller than chunk size by the overlap amount.

2. Step 2: Check the range step in options

   chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)] uses chunk_size - overlap as step, correctly creating overlaps.

3. Final Answer:

   chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)] -> Option C

4. Quick Check:

   Overlap step = chunk_size - overlap [OK]

Hint: Overlap step = chunk size minus overlap length [OK]

Common Mistakes:

• Using chunk_size as step (no overlap)
• Using overlap as step (too small steps)
• Starting range at overlap instead of zero

3. Given text = 'abcdefghij', chunk_size = 4, and overlap = 2, what is the output of this code?

chunks = [text[i:i+chunk_size] for i in range(0, len(text)-overlap, chunk_size - overlap)]
print(chunks)

medium

A. ['abcd', 'cdef', 'efgh', 'ghij']

B. ['abcd', 'efgh', 'ij']

C. ['abcd', 'bcde', 'cdef', 'defg']

D. ['abcd', 'bcdf', 'cdeg', 'defh']

Solution

1. Step 1: Calculate step size

   Step = chunk_size - overlap = 4 - 2 = 2.

2. Step 2: Generate chunks using step 2

   Chunks are:
   i=0: text[0:4] = 'abcd'
   i=2: text[2:6] = 'cdef'
   i=4: text[4:8] = 'efgh'
   i=6: text[6:10] = 'ghij'

3. Final Answer:

   ['abcd', 'cdef', 'efgh', 'ghij'] -> Option A

4. Quick Check:

   Chunks overlap by 2 chars = ['abcd', 'cdef', 'efgh', 'ghij'] [OK]

Hint: Step = chunk size minus overlap; slice text accordingly [OK]

Common Mistakes:

• Ignoring overlap and stepping by chunk size
• Wrong slicing indices
• Confusing overlap with chunk size

4. This code aims to chunk text with overlap but has a bug:

chunk_size = 5
overlap = 2
chunks = []
for i in range(0, len(text), chunk_size + overlap):
    chunks.append(text[i:i+chunk_size])
print(chunks)

What is the error?

medium

A. Step size should be chunk_size - overlap, not chunk_size + overlap

B. Chunk size should be increased by overlap

C. Overlap should be zero for chunking

D. The loop should start at overlap, not zero

Solution

1. Step 1: Understand step size for overlapping chunks

   To create overlap, step size must be less than chunk size by overlap amount.

2. Step 2: Identify incorrect step in code

   Code uses chunk_size + overlap which skips overlap, causing gaps.

3. Final Answer:

   Step size should be chunk_size - overlap, not chunk_size + overlap -> Option A

4. Quick Check:

   Overlap step = chunk_size - overlap [OK]

Hint: Overlap step = chunk size minus overlap, not plus [OK]

Common Mistakes:

• Adding overlap instead of subtracting
• Setting overlap to zero incorrectly
• Changing loop start index wrongly

5. You have a very long document and want to chunk it for an AI model. You want each chunk to have 100 words and overlap by 20 words to keep context. Which strategy balances chunk size and context best?

hard

A. Use chunk size 80 and step size 100 to create non-overlapping chunks

B. Use chunk size 100 and step size 100 to create overlapping chunks

C. Use chunk size 120 and step size 100 to create overlapping chunks

D. Use chunk size 100 and step size 80 (100 - 20) to create overlapping chunks

Solution

1. Step 1: Define chunk and step sizes for overlap

   Chunk size is 100 words, overlap is 20 words, so step size = 100 - 20 = 80.

2. Step 2: Choose correct step size to maintain overlap

   Step size 80 means each chunk starts 80 words after previous, overlapping 20 words.

3. Final Answer:

   Use chunk size 100 and step size 80 (100 - 20) to create overlapping chunks -> Option D

4. Quick Check:

   Step = chunk size - overlap = 80 [OK]

Hint: Step size = chunk size minus overlap for best context [OK]

Common Mistakes:

• Using step size larger than chunk size
• Setting overlap to zero accidentally
• Confusing chunk size with step size