Text splitters in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Text splitters

This pipeline breaks long text into smaller pieces so a model can understand and work with it better. It splits text by sentences or paragraphs before processing.

Data Flow - 4 Stages

1Raw Text Input

1 document x 1000 words→Receive full text document→1 document x 1000 words

"Today is sunny. We will go to the park. Then have lunch."

↓

2Sentence Splitter

1 document x 1000 words→Split text into sentences using punctuation→3 sentences x variable length

["Today is sunny.", "We will go to the park.", "Then have lunch."]

↓

3Paragraph Splitter

1 document x 1000 words→Split text into paragraphs by newline characters→3 paragraphs x variable length

["Today is sunny. We will go to the park.", "Then have lunch.", "After that, we will read books."]

↓

4Chunk Creation

10 sentences or 3 paragraphs→Group sentences or paragraphs into chunks of max 100 words→5 chunks x up to 100 words each

["Today is sunny. We will go to the park.", "Then have lunch. After that, we will read books."]

Training Trace - Epoch by Epoch

Loss
0.5 |****
0.4 |*** 
0.3 |**  
0.2 |*   
0.1 |    
     +----
      1 2 3 4 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.45	0.60	Initial split quality is moderate, some sentences split incorrectly.
2	0.30	0.75	Improved splitting rules reduce errors, better sentence boundaries.
3	0.20	0.85	Splitting is mostly correct, chunk sizes optimized for model input.
4	0.15	0.90	Final tuning reduces overlap and preserves context well.

Prediction Trace - 3 Layers

Layer 1: Input Raw Text

Layer 2: Sentence Splitter

Layer 3: Chunk Creation

Model Quiz - 3 Questions

What is the main purpose of the sentence splitter stage?

A To combine sentences into paragraphs

B To break text into smaller sentences for easier processing

C To remove punctuation from the text

D To translate text into another language

Key Insight

Splitting text into smaller, meaningful pieces helps models understand context better and improves processing efficiency. Training improves the splitting rules to reduce errors and optimize chunk sizes.