Computer use agents in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Computer use agents

Which metric matters for Computer use agents and WHY

For computer use agents, the key metrics are Precision and Recall. These agents often decide actions based on user commands or environmental data.

Precision tells us how often the agent's actions are correct when it decides to act. High precision means fewer wrong actions, which is important to avoid annoying or harmful mistakes.

Recall tells us how many of the correct actions the agent actually performs out of all possible correct actions. High recall means the agent does not miss important tasks.

Balancing these two helps ensure the agent acts correctly and does not miss important user needs.

Confusion Matrix for Computer use agents

      | Predicted Action | No Action |
      |------------------|-----------|
      | Action           | TP = 80   | FP = 20 |
      | No Action        | FN = 10   | TN = 90 |

      Total samples = 80 + 20 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 80 / (80 + 20) = 0.8
      Recall = TP / (TP + FN) = 80 / (80 + 10) = 0.8889
    

Precision vs Recall Tradeoff with Examples

If the agent is too cautious and only acts when very sure, it will have high precision but low recall. This means it rarely makes mistakes but may miss many tasks.

If the agent acts on many signals, it will have high recall but low precision. It does many tasks but also makes more mistakes.

Example: A smart assistant that controls home devices should avoid turning off lights wrongly (high precision) but also should not miss turning off lights when asked (high recall).

Good vs Bad Metric Values for Computer use agents

Good: Precision and recall both above 0.8 means the agent acts correctly most of the time and misses few tasks.

Bad: Precision below 0.5 means many wrong actions, annoying the user. Recall below 0.5 means many missed tasks, making the agent unreliable.

Common Pitfalls in Metrics for Computer use agents

• Accuracy paradox: If most times the agent does nothing, accuracy can be high even if it never acts correctly.
• Data leakage: Training on future user commands can inflate metrics unrealistically.
• Overfitting: Agent performs well on training data but poorly on new users or environments.

Self Check

Your computer use agent has 98% accuracy but only 12% recall on important user commands. Is it good for production?

Answer: No. The agent misses 88% of important commands, so it is unreliable despite high accuracy. It likely does nothing most of the time, inflating accuracy. Improving recall is critical.

Key Result

Precision and recall are key to ensure computer use agents act correctly and do not miss important tasks.

Practice

1. What is the main role of a computer use agent?

easy

A. To display graphics on the screen

B. To perform tasks automatically by sensing and acting

C. To store large amounts of data

D. To manually control the computer hardware

Solution

1. Step 1: Understand what an agent does

   An agent senses its environment and takes actions to complete tasks automatically.

2. Step 2: Compare options with this definition

   Only To perform tasks automatically by sensing and acting describes automatic task performance by sensing and acting.

3. Final Answer:

   To perform tasks automatically by sensing and acting -> Option B

4. Quick Check:

   Agent role = automatic task performance [OK]

Hint: Agents act automatically by sensing environment [OK]

Common Mistakes:

• Confusing agents with hardware controllers
• Thinking agents only store data
• Assuming agents only display information

2. Which of the following is the correct way to describe an agent's action cycle?

easy

A. Sense environment -> Take action -> Update environment

B. Take action -> Sense environment -> Sleep

C. Sense environment -> Sleep -> Take action

D. Update environment -> Take action -> Sense environment

Solution

1. Step 1: Recall the agent cycle steps

   An agent first senses its environment, then takes an action based on that sensing.

2. Step 2: Match the correct sequence

   Sense environment -> Take action -> Update environment correctly shows sensing first, then acting, then environment update.

3. Final Answer:

   Sense environment -> Take action -> Update environment -> Option A

4. Quick Check:

   Agent cycle = sense then act [OK]

Hint: Agents sense first, then act, then update [OK]

Common Mistakes:

• Mixing order of sensing and acting
• Including sleep incorrectly in cycle
• Ignoring environment update step

3. Consider this simple agent code snippet:

class Agent:
    def __init__(self):
        self.state = 0
    def sense(self, input):
        self.state += input
    def act(self):
        return self.state * 2

agent = Agent()
agent.sense(3)
agent.sense(4)
print(agent.act())

What is the output of this code?

medium

A. 14

B. 7

C. 12

D. 0

Solution

1. Step 1: Calculate state after sensing inputs

   Initial state is 0. After agent.sense(3), state = 3. After agent.sense(4), state = 7.

2. Step 2: Calculate action output

   agent.act() returns state * 2 = 7 * 2 = 14.

3. Final Answer:

   14 -> Option A

4. Quick Check:

   State sum 7 x 2 = 14 [OK]

Hint: Add inputs then multiply by 2 for output [OK]

Common Mistakes:

• Multiplying inputs separately instead of sum
• Using only last input instead of sum
• Confusing state update logic

4. This agent code has a bug:

class Agent:
    def __init__(self):
        self.state = 0
    def sense(self, input):
        self.state = input
    def act(self):
        return self.state * 2

agent = Agent()
agent.sense(3)
agent.sense(4)
print(agent.act())

What is the bug and how to fix it?

medium

A. Bug: sense method missing; Fix: add sense method

B. Bug: act returns wrong value; Fix: return state + 2

C. Bug: state overwritten each sense; Fix: use += to accumulate

D. Bug: state not initialized; Fix: initialize state in act

Solution

1. Step 1: Identify the problem in sense method

   The sense method sets state = input, so previous state is lost on each call.

2. Step 2: Fix by accumulating inputs

   Change state = input to state += input to keep adding inputs.

3. Final Answer:

   Bug: state overwritten each sense; Fix: use += to accumulate -> Option C

4. Quick Check:

   Accumulate inputs with += fixes bug [OK]

Hint: Use += to add inputs, not = to overwrite [OK]

Common Mistakes:

• Thinking act method is wrong
• Adding sense method again unnecessarily
• Initializing state in wrong place

5. You want to design a smart agent that automatically adjusts room temperature based on sensor data. Which approach best fits this task?

hard

A. Use a simple reflex agent that acts only on current sensor reading

B. Use a fixed schedule agent ignoring sensor data

C. Use a random agent that changes temperature randomly

D. Use a model-based agent that keeps track of past temperatures

Solution

1. Step 1: Understand task needs

   Adjusting temperature smartly requires remembering past data to avoid sudden changes.

2. Step 2: Choose agent type

   A model-based agent keeps track of past states, making it suitable for this task.

3. Final Answer:

   Use a model-based agent that keeps track of past temperatures -> Option D

4. Quick Check:

   Smart adjustment needs model-based agent [OK]

Hint: Smart agents remember past data for better decisions [OK]

Common Mistakes:

• Choosing simple reflex agent ignoring history
• Using random or fixed schedule agents
• Not considering past sensor data