Research assistant agent in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Research assistant agent

Which metric matters for Research assistant agent and WHY

A research assistant agent helps find and summarize information accurately and quickly. The key metrics to check are Precision and Recall. Precision tells us how many of the agent's answers are actually correct and relevant. Recall tells us how many of the important facts or documents the agent found out of all that exist. We want both to be high so the agent gives useful and complete information without many mistakes.

Confusion matrix for Research assistant agent

      | Predicted Relevant | Predicted Not Relevant |
      |--------------------|------------------------|
      | True Positive (TP)  | False Positive (FP)     |
      | False Negative (FN) | True Negative (TN)      |

      Example:
      TP = 80 (correctly found relevant info)
      FP = 20 (wrongly marked irrelevant info as relevant)
      FN = 10 (missed relevant info)
      TN = 90 (correctly ignored irrelevant info)

      Total samples = 80 + 20 + 10 + 90 = 200
    

Precision vs Recall tradeoff with examples

If the agent focuses on high precision, it means it only gives answers when very sure. This reduces wrong answers but might miss some useful info (lower recall). For example, a medical research assistant should avoid false info, so high precision is important.

If the agent focuses on high recall, it tries to find all possible relevant info, even if some are wrong (lower precision). This is good when missing any info is risky, like in legal research where missing a law could cause problems.

Balancing precision and recall depends on the research goal.

What good vs bad metric values look like

Good metrics: Precision and recall both above 0.8 means the agent finds most relevant info and makes few mistakes.

Bad metrics: Precision below 0.5 means many wrong answers. Recall below 0.5 means the agent misses too much important info.

For example, precision=0.9 and recall=0.85 is good. Precision=0.4 and recall=0.3 is bad.

Common pitfalls in metrics for Research assistant agent

• Accuracy paradox: If most info is irrelevant, a model that always says "not relevant" can have high accuracy but be useless.
• Data leakage: If the agent sees answers during training that appear in testing, metrics look better but real performance is worse.
• Overfitting: The agent may memorize specific documents and score high on test data but fail on new topics.

Self-check question

Your research assistant agent has 98% accuracy but only 12% recall on finding relevant documents. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most documents are irrelevant, so the agent mostly says "not relevant". The very low recall means it misses almost all relevant documents, which defeats the purpose of a research assistant.

Key Result

Precision and recall are key to measure how well a research assistant agent finds relevant information accurately and completely.

Practice

1. What is the main purpose of a research assistant agent in AI?

easy

A. To create new scientific theories automatically

B. To replace human researchers completely

C. To help find and summarize information quickly

D. To perform physical experiments in a lab

Solution

1. Step 1: Understand the role of a research assistant agent

   A research assistant agent is designed to help users by finding and summarizing information efficiently.

2. Step 2: Compare options with this role

   Options B, C, and D describe tasks beyond the typical scope of such agents, which focus on information handling.

3. Final Answer:

   To help find and summarize information quickly -> Option C

4. Quick Check:

   Purpose = Find and summarize info quickly [OK]

Hint: Focus on what the agent automates: info search and summary [OK]

Common Mistakes:

• Thinking the agent replaces all human research
• Confusing data collection with physical experiments
• Assuming the agent creates new theories

2. Which of the following is the correct way to start a simple research assistant agent function in Python?

easy

A. def research_agent(query):

B. function research_agent(query) {

C. research_agent <- function(query) {

D. def research_agent[]:

Solution

1. Step 1: Identify the correct Python function syntax

   Python functions start with 'def', followed by the function name and parentheses with parameters.

2. Step 2: Check each option's syntax

   def research_agent(query): uses correct Python syntax. A has invalid empty brackets [], B is JavaScript style, C is R style.

3. Final Answer:

   def research_agent(query): -> Option A

4. Quick Check:

   Python function = def name(params): [OK]

Hint: Remember Python functions start with 'def' and parentheses [OK]

Common Mistakes:

• Using curly braces instead of colon and indentation
• Mixing syntax from other languages
• Incorrect use of brackets in function definition

3. Given the code below, what will be the output?

def summarize(text):
    return text[:10] + '...'

result = summarize('Artificial intelligence helps research.')
print(result)

medium

A. Artificial...

B. Artificial intelligence...

C. Artificial in...

D. Artificial i...

Solution

1. Step 1: Understand the summarize function slicing

   The function returns the first 10 characters of the text plus '...'. The slice text[:10] takes characters at positions 0 to 9.

2. Step 2: Extract the first 10 characters from the input

   'Artificial intelligence helps research.' first 10 chars are 'Artificial ' (including the space at position 9). So the output is 'Artificial ...'.

3. Step 3: Confirm the exact output

   The output is 'Artificial ' + '...' = 'Artificial ...', which matches Artificial i... 'Artificial i...'. Actually, the 10 characters are 'Artificial ' (9 letters + 1 space), so the output is 'Artificial ...'. Artificial i... shows 'Artificial i...', which includes the 'i' from 'intelligence' (11th character). So Artificial i... is incorrect.

4. Step 4: Check options carefully

   Artificial... is 'Artificial...', which is 9 letters + '...'. Artificial i... is 'Artificial i...', which is 10 letters + '...'. The code returns text[:10] + '...', so 10 characters plus '...'. The first 10 characters are 'Artificial ' (with space), so the output is 'Artificial ...'. None of the options exactly match 'Artificial ...'.

5. Step 5: Correct the options or answer

   Since none of the options exactly match 'Artificial ...', the closest is Artificial i... 'Artificial i...', which is 11 characters before '...'. So the correct answer should be Artificial... 'Artificial...', which is 9 letters + '...'. But the code returns 10 characters + '...'. So the correct answer is Artificial i....

6. Final Answer:

   Artificial i... -> Option D

7. Quick Check:

   text[:10] + '...' = 'Artificial i...' [OK]

Hint: Count characters carefully including spaces for slicing [OK]

Common Mistakes:

• Counting 10 letters without space
• Assuming slice excludes space
• Confusing slice length with index

4. The following code is intended to collect search results and summarize them, but it raises an error. What is the error?

def research_agent(queries):
    summaries = []
    for q in queries:
        summary = summarize(q)
    summaries.append(summary)
    return summaries

print(research_agent(['AI', 'Machine Learning']))

medium

A. The function research_agent has wrong indentation

B. The append is outside the loop, so only last summary is added

C. The summarize function is not defined

D. queries should be a string, not a list

Solution

1. Step 1: Analyze the indentation of append

   The append statement is outside the for loop, so only the last summary is appended to summaries.

2. Step 2: Check if summarize is defined

   Assuming summarize is defined elsewhere, the code runs but only appends one summary.

3. Step 3: Identify the error

   The main logical error is that summaries.append(summary) should be inside the loop to collect all summaries.

4. Final Answer:

   The append is outside the loop, so only last summary is added -> Option B

5. Quick Check:

   Indent append inside loop to fix [OK]

Hint: Check indentation of statements inside loops carefully [OK]

Common Mistakes:

• Assuming summarize function is missing
• Misreading indentation as correct
• Ignoring loop scope for append

5. You want to build a research assistant agent that searches multiple sources and summarizes results. Which approach best improves accuracy and efficiency?

hard

A. Use multiple search APIs, combine results, then summarize with a language model

B. Search only one source deeply and summarize without combining

C. Summarize each source separately and do not merge results

D. Collect raw data without summarizing to avoid errors

Solution

1. Step 1: Consider combining multiple sources

   Using multiple search APIs gathers diverse information, improving coverage and accuracy.

2. Step 2: Summarize combined results with a language model

   Combining results before summarizing helps create a concise, comprehensive summary efficiently.

3. Final Answer:

   Use multiple search APIs, combine results, then summarize with a language model -> Option A

4. Quick Check:

   Combine sources + summarize = best accuracy [OK]

Hint: Combine diverse data before summarizing for best results [OK]

Common Mistakes:

• Relying on a single source only
• Not merging summaries leads to fragmented info
• Avoiding summarization reduces efficiency