Text splitters in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a text splitter in the context of machine learning?

A text splitter is a tool or method that breaks long text into smaller parts or chunks. This helps models handle text better by focusing on manageable pieces instead of very long text all at once.

beginner

Why do we need to split text before feeding it to AI models?

Splitting text helps avoid overwhelming the model with too much information at once. It also helps keep context clear and allows the model to process and understand text more effectively.

beginner

Name two common ways to split text.

1. Splitting by sentences or paragraphs. 2. Splitting by fixed length chunks (like every 100 words). Both help keep text pieces easy to handle.

intermediate

What is an overlap in text splitting and why is it useful?

Overlap means some words from the end of one chunk appear again at the start of the next chunk. This helps keep context between chunks so the model doesn’t lose important connections.

intermediate

How does chunk size affect model performance?

If chunks are too small, the model might miss context. If too large, the model might get overwhelmed or run into limits. Choosing the right chunk size balances understanding and performance.

What is the main purpose of using a text splitter?

A To remove punctuation from text

B To translate text into another language

C To break long text into smaller parts for easier processing

D To generate new text from old text

Text splitters break long text into smaller chunks so models can handle it better.

Which of these is a common way to split text?

A By sentence or paragraph

B By font size

C By color of words

D By word meaning

Splitting by sentence or paragraph is a common and natural way to divide text.

What does overlap in text splitting help with?

A Changing the text language

B Adding more words to the text

C Removing repeated words

D Keeping context between chunks

Overlap repeats some words between chunks to keep context clear.

What happens if text chunks are too large?

A Model might get overwhelmed or hit limits

B Model will understand better

C Text becomes shorter

D Text becomes easier to read

Very large chunks can overwhelm the model or exceed its input limits.

Why not make chunks too small?

A Small chunks confuse the model

B Small chunks can lose important context

C Small chunks increase text length

D Small chunks are harder to process

If chunks are too small, the model might miss important connections in the text.

Explain what a text splitter is and why it is important in AI text processing.

Think about how breaking text into smaller parts helps AI understand better.

Describe how overlap works in text splitting and why it helps maintain context.

Overlap is like sharing some words between chunks to keep the story connected.