Multi-query retrieval in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Multi-query retrieval

Multi-query retrieval is a process where multiple questions or queries are used together to find the best matching information from a large collection of data. It helps improve search results by combining different queries to get more accurate answers.

Data Flow - 6 Stages

1Input Queries

5 queries→Receive multiple queries from user→5 queries

["What is AI?", "Explain machine learning", "Benefits of AI", "AI applications", "Future of AI"]

↓

2Query Embedding

5 queries→Convert each query into a vector representation→5 vectors of size 128

[[0.12, 0.34, ..., 0.56], [0.22, 0.11, ..., 0.44], ...]

↓

3Document Embedding

10000 documents→Convert documents into vector representations→10000 vectors of size 128

[[0.10, 0.30, ..., 0.50], [0.20, 0.10, ..., 0.40], ...]

↓

4Similarity Calculation

5 query vectors, 10000 document vectors→Calculate similarity scores between each query and all documents→5 arrays of 10000 similarity scores

[[0.8, 0.1, ..., 0.3], [0.7, 0.2, ..., 0.4], ...]

↓

5Score Aggregation

5 arrays of 10000 scores→Combine scores from all queries for each document→1 array of 10000 aggregated scores

[0.85, 0.25, ..., 0.45]

↓

6Ranking and Retrieval

1 array of 10000 scores→Sort documents by aggregated score and select top results→Top 10 documents

[DocID 123, DocID 456, ..., DocID 789]

Training Trace - Epoch by Epoch

Epoch 1: ******
Epoch 2: **********
Epoch 3: **************
Epoch 4: ****************
Epoch 5: ******************
(Loss decreasing, accuracy increasing)

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.55	Model starts learning to match queries and documents
2	0.48	0.68	Loss decreases and accuracy improves as embeddings get better
3	0.35	0.78	Model shows good convergence with improved retrieval quality
4	0.28	0.83	Further refinement of embeddings and similarity scoring
5	0.22	0.87	Training stabilizes with high accuracy and low loss

Prediction Trace - 5 Layers

Layer 1: Input Queries

Layer 2: Query Embedding

Layer 3: Similarity Calculation

Layer 4: Score Aggregation

Layer 5: Ranking and Retrieval

Model Quiz - 3 Questions

Why do we convert queries and documents into vectors?

A To make them readable by humans

B To compare them mathematically

C To reduce their size

D To encrypt the data

Key Insight

Multi-query retrieval improves search by combining multiple questions to better match documents. Embedding queries and documents into vectors allows easy comparison. Training reduces loss and increases accuracy, making retrieval more precise.