Sentence transformers in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Sentence transformers

This pipeline converts sentences into numbers that computers understand well. It helps find how similar sentences are or groups sentences by meaning.

Data Flow - 4 Stages

1Input sentences

1000 sentences→Raw text input→1000 sentences

"I love apples.", "The sky is blue."

↓

2Tokenization

1000 sentences→Split sentences into words or pieces→1000 lists of tokens

[['I', 'love', 'apples', '.'], ['The', 'sky', 'is', 'blue', '.']]

↓

3Embedding generation

1000 lists of tokens→Convert tokens into 768-dimensional vectors using transformer model→1000 lists of token vectors x 768 dimensions

[[0.12, -0.05, ..., 0.33], [0.07, 0.01, ..., -0.22]]

↓

4Pooling

1000 lists of token vectors x 768 dimensions→Combine token vectors into one sentence vector→1000 vectors x 768 dimensions

[[0.10, -0.03, ..., 0.30], [0.05, 0.00, ..., -0.20]]

Training Trace - Epoch by Epoch

Loss
0.9 |*       
0.8 | **     
0.7 |  **    
0.6 |   **   
0.5 |    **  
0.4 |     ** 
0.3 |      **
    +--------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.50	Model starts learning sentence meanings
2	0.65	0.65	Loss decreases, accuracy improves
3	0.50	0.75	Model better understands sentence similarity
4	0.40	0.82	Training converges with good accuracy
5	0.35	0.85	Final epoch with best performance

Prediction Trace - 4 Layers

Layer 1: Input sentence

Layer 2: Tokenization

Layer 3: Transformer embedding

Layer 4: Pooling

Model Quiz - 3 Questions

What does the pooling step do in the sentence transformer pipeline?

A Splits sentences into words

B Combines token vectors into one sentence vector

C Converts sentences to raw text

D Calculates loss during training

Key Insight

Sentence transformers turn sentences into fixed-size number lists that capture meaning. This helps computers compare sentences easily and find similar ones.