Text chunking strategies in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is text chunking in natural language processing?

Text chunking is the process of dividing text into smaller, meaningful pieces called chunks, such as phrases or sentences, to make it easier for machines to understand and analyze.

beginner

Name two common strategies used for text chunking.

Two common strategies are:<br>1. Fixed-size chunking: splitting text into equal-sized parts.<br>2. Semantic chunking: splitting text based on meaning, like sentences or phrases.

intermediate

Why is semantic chunking often better than fixed-size chunking?

Semantic chunking respects the meaning and structure of text, like sentences or paragraphs, which helps models understand context better than arbitrary fixed-size chunks.

intermediate

What is a challenge when using fixed-size chunking?

Fixed-size chunking can split sentences or ideas in the middle, causing loss of meaning and making it harder for models to understand the text properly.

advanced

How can overlapping chunks improve text chunking?

Overlapping chunks include some shared text between chunks, which helps preserve context across chunks and reduces information loss at chunk boundaries.

What does text chunking help with in machine learning?

A Generating images from text

B Translating text into another language

C Breaking text into smaller, meaningful parts

D Encrypting text data

Text chunking breaks text into smaller parts to help machines understand and analyze it better.

Which chunking strategy respects sentence boundaries?

A Semantic chunking

B Fixed-size chunking

C Random chunking

D No chunking

Semantic chunking splits text based on meaning, like sentences, preserving natural boundaries.

What is a downside of fixed-size chunking?

A It can split sentences in the middle

B It always preserves sentence meaning

C It requires complex algorithms

D It only works for images

Fixed-size chunking may cut sentences arbitrarily, losing meaning.

Why use overlapping chunks?

A To reduce chunk size

B To preserve context across chunks

C To speed up processing

D To remove stop words

Overlapping chunks share text to keep context between chunks.

Which is NOT a text chunking strategy?

A Semantic chunking

B Fixed-size chunking

C Random chunking

D Image chunking

Image chunking is unrelated to text chunking.

Explain what text chunking is and why it is useful in natural language processing.

Think about how breaking text into parts helps computers.

Describe the difference between fixed-size chunking and semantic chunking, including one advantage and one disadvantage of each.

Consider how chunks are created and how meaning is preserved.