Measuring agent accuracy and relevance in Agentic AI - Model Pipeline Trace

Model Pipeline - Measuring agent accuracy and relevance

This pipeline shows how an agent learns to give accurate and relevant answers. It starts with data, processes it, trains the agent, and checks how well it performs.

Data Flow - 6 Stages

1Raw input data

1000 rows x 3 columns→Collect user queries, correct answers, and context→1000 rows x 3 columns

Query: 'What is AI?', Answer: 'Artificial Intelligence', Context: 'Technology basics'

↓

2Preprocessing

1000 rows x 3 columns→Clean text, remove noise, and standardize format→1000 rows x 3 columns

Query: 'what is AI?', Answer: 'artificial intelligence', Context: 'technology basics'

↓

3Feature extraction

1000 rows x 3 columns→Convert text to numerical vectors using embeddings→1000 rows x 300 columns

Query vector: [0.12, -0.05, ..., 0.33]

↓

4Train/test split

1000 rows x 300 columns→Split data into training (80%) and testing (20%) sets→Train: 800 rows x 300 columns, Test: 200 rows x 300 columns

Training query vector sample and label

↓

5Model training

800 rows x 300 columns→Train agent model to predict correct answers→Trained model

Model learns to map query vectors to answers

↓

6Evaluation

200 rows x 300 columns→Test model predictions and calculate accuracy and relevance→Accuracy score, Relevance score

Accuracy: 0.85, Relevance: 0.80

Training Trace - Epoch by Epoch

Loss
0.7 |*
0.6 |** 
0.5 |***  
0.4 |****   
0.3 |*****    
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning, accuracy is low
2	0.50	0.70	Loss decreases, accuracy improves
3	0.40	0.78	Model learns relevant patterns
4	0.32	0.83	Accuracy continues to rise
5	0.28	0.85	Training converges with good accuracy

Prediction Trace - 5 Layers

Layer 1: Input query vector

Layer 2: Model embedding layer

Layer 3: Prediction layer

Layer 4: Softmax normalization

Layer 5: Final answer selection

Model Quiz - 3 Questions

What does the loss value represent during training?

A How wrong the agent's predictions are

B The number of correct answers

C The size of the input data

D The speed of training

Key Insight

Measuring both accuracy and relevance helps ensure the agent not only answers correctly but also provides answers that fit the user's question context well.

Practice

1. What does accuracy measure when evaluating an AI agent's answers?

easy

A. How many answers are related but not exact

B. How fast the agent responds

C. How many answers are exactly correct

D. How many answers are generated

Solution

1. Step 1: Understand accuracy definition

   Accuracy counts the number of answers that match the correct ones exactly.

2. Step 2: Compare with other metrics

   Relevance measures usefulness, not exact correctness, so it is different from accuracy.

3. Final Answer:

   How many answers are exactly correct -> Option C

4. Quick Check:

   Accuracy = exact correctness [OK]

Hint: Accuracy means exact right answers only [OK]

Common Mistakes:

• Confusing accuracy with relevance
• Thinking accuracy measures speed
• Assuming accuracy counts all related answers

2. Which of the following is the correct way to calculate accuracy for an AI agent's answers?

easy

A. Number of related answers divided by total answers

B. Number of correct answers divided by total answers

C. Number of answers generated per second

D. Number of answers ignored by the agent

Solution

1. Step 1: Recall accuracy formula

   Accuracy = (correct answers) / (total answers given).

2. Step 2: Eliminate incorrect options

   Options about related answers or speed do not define accuracy.

3. Final Answer:

   Number of correct answers divided by total answers -> Option B

4. Quick Check:

   Accuracy = correct / total [OK]

Hint: Accuracy = correct answers ÷ total answers [OK]

Common Mistakes:

• Using related answers count instead of correct
• Mixing speed with accuracy
• Ignoring total number of answers

3. Given an AI agent answered 80 questions, 60 were exactly correct, and 10 more were relevant but not exact. What is the accuracy and relevance percentage?

medium

A. Accuracy 60%, Relevance 70%

B. Accuracy 60%, Relevance 87.5%

C. Accuracy 75%, Relevance 60%

D. Accuracy 75%, Relevance 87.5%

Solution

1. Step 1: Calculate accuracy percentage

   Accuracy = (60 correct / 80 total) * 100 = 75%.

2. Step 2: Calculate relevance percentage

   Relevance = ((60 correct + 10 relevant) / 80 total) * 100 = 87.5%.

3. Final Answer:

   Accuracy 75%, Relevance 87.5% -> Option D

4. Quick Check:

   Accuracy = 75%, Relevance = 87.5% [OK]

Hint: Add relevant to correct for relevance % [OK]

Common Mistakes:

• Mixing accuracy and relevance values
• Not adding relevant answers for relevance
• Dividing by wrong total number

4. An AI agent evaluation code snippet is below. It calculates accuracy but returns 0. What is the bug?

correct = 50
total = 0
accuracy = correct / total
print(accuracy)

medium

A. Division by zero error due to total being zero

B. Correct variable is zero, so accuracy is zero

C. Print statement syntax is wrong

D. Accuracy should be multiplied by 100

Solution

1. Step 1: Identify variables and operation

   correct = 50, total = 0, accuracy = correct / total.

2. Step 2: Check for division errors

   Dividing by zero (total=0) causes an error or invalid result.

3. Final Answer:

   Division by zero error due to total being zero -> Option A

4. Quick Check:

   Division by zero causes error [OK]

Hint: Check denominator is not zero before dividing [OK]

Common Mistakes:

• Ignoring zero division error
• Thinking print syntax is wrong
• Assuming accuracy must be multiplied by 100

5. You want to improve an AI agent's trust by measuring both accuracy and relevance. Which approach best helps achieve this?

hard

A. Track exact correct answers and also count useful related answers

B. Only count answers that are exactly correct

C. Ignore relevance and focus on speed of answers

D. Count all answers regardless of correctness or relevance

Solution

1. Step 1: Understand trust factors

   Trust improves when answers are both correct and useful (relevant).

2. Step 2: Choose measurement approach

   Tracking both exact correctness (accuracy) and usefulness (relevance) gives a fuller picture.

3. Final Answer:

   Track exact correct answers and also count useful related answers -> Option A

4. Quick Check:

   Measure accuracy + relevance for trust [OK]

Hint: Measure both exact and useful answers for trust [OK]

Common Mistakes:

• Focusing only on exact correctness
• Ignoring relevance completely
• Measuring speed instead of quality