Measuring agent accuracy and relevance in Agentic AI - Model Pipeline Trace

Model Pipeline - Measuring agent accuracy and relevance

This pipeline shows how an agent learns to give accurate and relevant answers. It starts with data, processes it, trains the agent, and checks how well it performs.

Data Flow - 6 Stages

1Raw input data

1000 rows x 3 columns→Collect user queries, correct answers, and context→1000 rows x 3 columns

Query: 'What is AI?', Answer: 'Artificial Intelligence', Context: 'Technology basics'

↓

2Preprocessing

1000 rows x 3 columns→Clean text, remove noise, and standardize format→1000 rows x 3 columns

Query: 'what is AI?', Answer: 'artificial intelligence', Context: 'technology basics'

↓

3Feature extraction

1000 rows x 3 columns→Convert text to numerical vectors using embeddings→1000 rows x 300 columns

Query vector: [0.12, -0.05, ..., 0.33]

↓

4Train/test split

1000 rows x 300 columns→Split data into training (80%) and testing (20%) sets→Train: 800 rows x 300 columns, Test: 200 rows x 300 columns

Training query vector sample and label

↓

5Model training

800 rows x 300 columns→Train agent model to predict correct answers→Trained model

Model learns to map query vectors to answers

↓

6Evaluation

200 rows x 300 columns→Test model predictions and calculate accuracy and relevance→Accuracy score, Relevance score

Accuracy: 0.85, Relevance: 0.80

Training Trace - Epoch by Epoch

Loss
0.7 |*
0.6 |** 
0.5 |***  
0.4 |****   
0.3 |*****    
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning, accuracy is low
2	0.50	0.70	Loss decreases, accuracy improves
3	0.40	0.78	Model learns relevant patterns
4	0.32	0.83	Accuracy continues to rise
5	0.28	0.85	Training converges with good accuracy

Prediction Trace - 5 Layers

Layer 1: Input query vector

Layer 2: Model embedding layer

Layer 3: Prediction layer

Layer 4: Softmax normalization

Layer 5: Final answer selection

Model Quiz - 3 Questions

What does the loss value represent during training?

A How wrong the agent's predictions are

B The number of correct answers

C The size of the input data

D The speed of training

Key Insight

Measuring both accuracy and relevance helps ensure the agent not only answers correctly but also provides answers that fit the user's question context well.