Autonomous web browsing agents in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Autonomous web browsing agents

Which metric matters for Autonomous web browsing agents and WHY

For autonomous web browsing agents, key metrics include task success rate, precision, and recall. Task success rate measures how often the agent completes the intended browsing task correctly, such as finding information or filling forms. Precision tells us how many of the agent's actions were correct out of all actions it took, avoiding unnecessary or wrong clicks. Recall shows how many needed actions the agent actually performed, ensuring it does not miss important steps. These metrics matter because the agent must be both accurate and thorough to be useful and safe.

Confusion matrix for agent actions

Actions Taken by Agent
+----------------+----------------+----------------+
|                | Action Correct | Action Wrong   |
+----------------+----------------+----------------+
| Action Needed  | True Positive  | False Negative |
| Action Not Needed | False Positive | True Negative  |
+----------------+----------------+----------------+

Where:
- TP: Agent correctly performed a needed action.
- FP: Agent performed an unnecessary or wrong action.
- FN: Agent missed a needed action.
- TN: Agent correctly avoided unnecessary actions.

Metrics:
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
Task Success Rate = Number of tasks completed correctly / Total tasks
    

Precision vs Recall tradeoff with examples

If the agent has high precision but low recall, it means it rarely makes wrong moves but often misses important steps. For example, it clicks only when very sure but may skip filling some form fields, causing incomplete tasks.

If the agent has high recall but low precision, it tries to do all needed actions but also many wrong ones. For example, it clicks many buttons, including irrelevant ones, which may cause errors or slow performance.

Balancing precision and recall is important: the agent should do all necessary actions (high recall) but avoid mistakes (high precision) to complete tasks efficiently and correctly.

What "good" vs "bad" metric values look like for Autonomous web browsing agents

• Good: Task success rate above 90%, precision and recall both above 85%. The agent completes tasks reliably with few mistakes or missed steps.
• Bad: Task success rate below 60%, precision or recall below 50%. The agent often fails tasks, clicks wrong elements, or misses important actions.

Common pitfalls in metrics for Autonomous web browsing agents

• Accuracy paradox: High overall accuracy can be misleading if the agent mostly does nothing and avoids errors but also never completes tasks.
• Data leakage: Training the agent on test websites can inflate metrics but fail in real browsing scenarios.
• Overfitting: Agent performs well on known sites but poorly on new or dynamic pages.
• Ignoring user experience: Metrics may not capture delays or confusing agent behavior that frustrates users.

Self-check question

Your autonomous web browsing agent has 98% accuracy but only 12% recall on needed actions. Is it good for production? Why or why not?

Answer: No, it is not good. The agent rarely makes mistakes (high accuracy) but misses most needed actions (very low recall). This means it often fails to complete tasks, making it unreliable despite high accuracy.

Key Result

For autonomous web browsing agents, balancing high precision and recall ensures reliable task completion without unnecessary actions.

Practice

1. What is the main purpose of an autonomous web browsing agent?

easy

A. To automatically explore and interact with websites without human help

B. To manually browse websites faster

C. To replace web servers

D. To create websites from scratch

Solution

1. Step 1: Understand the role of autonomous agents

   They act automatically to perform tasks on websites without needing a person to control them.

2. Step 2: Compare options with this role

   Only To automatically explore and interact with websites without human help describes automatic exploration and interaction, which matches the agent's purpose.

3. Final Answer:

   To automatically explore and interact with websites without human help -> Option A

4. Quick Check:

   Autonomous means automatic = B [OK]

Hint: Autonomous means automatic, so pick the automatic action [OK]

Common Mistakes:

• Confusing manual browsing with autonomous
• Thinking agents create websites
• Assuming agents replace servers

2. Which of the following is the correct syntax to make an autonomous agent click a button with id 'submit'?

easy

A. agent.click('#submit')

B. agent.clickById('submit')

C. agent.pressButton('submit')

D. agent.clickButton('#submit')

Solution

1. Step 1: Identify common syntax for clicking elements

   Most agents use a method like click with a CSS selector string, e.g., '#submit'.

2. Step 2: Check each option's method and argument

   agent.click('#submit') uses click with '#submit', which is standard CSS selector syntax. Others use non-standard method names or incorrect selectors.

3. Final Answer:

   agent.click('#submit') -> Option A

4. Quick Check:

   Click method + CSS selector = D [OK]

Hint: Click uses CSS selectors like '#id' inside parentheses [OK]

Common Mistakes:

• Using method names not supported by agents
• Passing id without '#' selector
• Confusing click with pressButton

3. Given this code snippet for an autonomous agent:

agent.goTo('https://example.com')
agent.fill('#name', 'Alice')
agent.click('#submit')
print(agent.getLastResponseStatus())

What will be printed if the form submission is successful?

medium

A. 404

B. 500

C. 200

D. 0

Solution

1. Step 1: Understand HTTP status codes

   200 means success, 404 means not found, 500 means server error, 0 means no response.

2. Step 2: Analyze the code's last line

   The method getLastResponseStatus() returns the HTTP status of the last request, which should be 200 if submission succeeded.

3. Final Answer:

   200 -> Option C

4. Quick Check:

   Success status code = 200 [OK]

Hint: Success HTTP status is 200, always remember [OK]

Common Mistakes:

• Confusing 404 (not found) with success
• Thinking 500 means success
• Assuming 0 means success

4. You wrote this code for an autonomous agent:

agent.goTo('https://example.com')
agent.fill('name', 'Bob')
agent.click('#submit')

But the form never submits. What is the likely error?

medium

A. Agent cannot fill forms automatically

B. Using 'goTo' instead of 'navigateTo'

C. Click method should be 'press' instead of 'click'

D. Missing '#' in the selector for the fill method

Solution

1. Step 1: Check the selector used in fill method

   The fill method expects a CSS selector. 'name' without '#' targets a tag, not an id.

2. Step 2: Understand impact of wrong selector

   Without '#', the agent cannot find the input field with id 'name', so filling fails and form won't submit.

3. Final Answer:

   Missing '#' in the selector for the fill method -> Option D

4. Quick Check:

   CSS id selectors need '#' prefix [OK]

Hint: Always prefix id selectors with '#' in fill and click [OK]

Common Mistakes:

• Thinking goTo is wrong method
• Believing click method is incorrect
• Assuming agents can't fill forms

5. You want an autonomous web browsing agent to collect all links on a page and visit only those that contain the word 'news'. Which approach is best?

hard

A. Use agent.click('a') to click all links on the page automatically

B. Use agent.getLinks() to get all links, filter those with 'news' in URL, then agent.goTo() each filtered link

C. Use agent.fill('input', 'news') to search for news links

D. Use agent.submit() without filtering links

Solution

1. Step 1: Identify how to get all links

   The method agent.getLinks() returns all links on the page as URLs.

2. Step 2: Filter links containing 'news' and visit them

   Filter the list for URLs containing 'news', then use agent.goTo() to visit each filtered link.

3. Final Answer:

   Use agent.getLinks() to get all links, filter those with 'news' in URL, then agent.goTo() each filtered link -> Option B

4. Quick Check:

   Get links + filter + visit = A [OK]

Hint: Get all links, filter by keyword, then visit filtered links [OK]

Common Mistakes:

• Clicking all links blindly without filtering
• Using fill to search links incorrectly
• Submitting without selecting links