RAG evaluation metrics in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - RAG evaluation metrics

This pipeline shows how a Retrieval-Augmented Generation (RAG) model processes input text by retrieving relevant documents, generating answers, and then evaluating the quality of those answers using specific metrics.

Data Flow - 4 Stages

1Input Text

1 sample x 1 text string→User provides a question or prompt→1 sample x 1 text string

"What is the capital of France?"

↓

2Document Retrieval

1 sample x 1 text string→Retrieve top relevant documents from knowledge base→1 sample x 5 documents (text)

["Paris is the capital of France.", "France is in Europe.", "The Eiffel Tower is in Paris.", "Paris has many museums.", "French cuisine is famous."]

↓

3Answer Generation

1 sample x 5 documents→Generate answer using retrieved documents and input question→1 sample x 1 generated answer string

"The capital of France is Paris."

↓

4Evaluation Metrics Calculation

1 sample x 1 generated answer string + 1 reference answer string→Calculate metrics like Exact Match, F1 Score, and Rouge-L→1 sample x 3 metric scores

{"Exact Match": 1.0, "F1 Score": 1.0, "ROUGE-L": 0.85}

Training Trace - Epoch by Epoch

Loss
1.0 |***************
0.8 |************   
0.6 |********      
0.4 |******        
0.2 |***           
0.0 +-------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.60	Initial training with moderate loss and accuracy.
2	0.65	0.72	Loss decreased, accuracy improved as model learns retrieval and generation.
3	0.50	0.80	Better alignment between retrieved documents and generated answers.
4	0.40	0.85	Model shows strong retrieval and generation performance.
5	0.35	0.88	Training converges with high accuracy and low loss.

Prediction Trace - 4 Layers

Layer 1: Input Question

Layer 2: Document Retrieval

Layer 3: Answer Generation

Layer 4: Evaluation Metrics Calculation

Model Quiz - 3 Questions

Which stage reduces the input from one question to multiple documents?

A Evaluation Metrics Calculation

B Answer Generation

C Document Retrieval

D Input Text

Key Insight

RAG models combine retrieval and generation steps, and their evaluation metrics like Exact Match and F1 Score help measure how well the generated answers match reference answers. Training shows steady improvement as the model learns to retrieve relevant documents and generate accurate responses.