Computer use agents in Agentic AI - Model Pipeline Trace

Model Pipeline - Computer use agents

This pipeline shows how a computer use agent learns to assist users by observing their actions and predicting helpful next steps. It starts with collecting user activity data, processes it, trains a model to understand patterns, and then makes predictions to support the user.

Data Flow - 6 Stages

1Data Collection

1000 sessions x 10 features→Collect user activity logs including clicks, typing speed, and app usage→1000 sessions x 10 features

Session 1: {clicks: 15, typing_speed: 40 wpm, app: browser, time_spent: 5 min, ...}

↓

2Preprocessing

1000 sessions x 10 features→Clean data, fill missing values, normalize numerical features→1000 sessions x 10 features

Typing speed normalized from 40 wpm to 0.8 (scale 0-1)

↓

3Feature Engineering

1000 sessions x 10 features→Create new features like average time between clicks and app switch frequency→1000 sessions x 12 features

Added features: avg_click_interval=2 sec, app_switches=3

↓

4Model Training

800 sessions x 12 features→Train agent model to predict next user action→Trained model

Model learns patterns to predict if user will open email next

↓

5Validation

200 sessions x 12 features→Evaluate model accuracy on unseen data→Accuracy score: 85%

Model correctly predicted next action in 170 out of 200 sessions

↓

6Prediction

1 session x 12 features→Agent predicts next helpful action for user→Prediction: Open calendar app

Based on current activity, agent suggests calendar to user

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |*   
0.3 |    
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning basic patterns
2	0.50	0.70	Loss decreases, accuracy improves
3	0.40	0.78	Model captures more user behavior details
4	0.32	0.83	Good convergence, model is reliable
5	0.28	0.85	Final epoch with stable performance

Prediction Trace - 5 Layers

Layer 1: Input Features

Layer 2: Neural Network Layer 1 (ReLU)

Layer 3: Neural Network Layer 2 (ReLU)

Layer 4: Output Layer (Softmax)

Layer 5: Decision

Model Quiz - 3 Questions

What happens to the data shape after feature engineering?

A It decreases in number of rows

B It stays the same

C It increases in number of columns

D It becomes a single number

Key Insight

This visualization shows how a computer use agent learns from user activity data by transforming raw inputs into meaningful features, training a model that improves over time, and making predictions that help users by suggesting their next likely action.

Practice

1. What is the main role of a computer use agent?

easy

A. To display graphics on the screen

B. To perform tasks automatically by sensing and acting

C. To store large amounts of data

D. To manually control the computer hardware

Solution

1. Step 1: Understand what an agent does

   An agent senses its environment and takes actions to complete tasks automatically.

2. Step 2: Compare options with this definition

   Only To perform tasks automatically by sensing and acting describes automatic task performance by sensing and acting.

3. Final Answer:

   To perform tasks automatically by sensing and acting -> Option B

4. Quick Check:

   Agent role = automatic task performance [OK]

Hint: Agents act automatically by sensing environment [OK]

Common Mistakes:

• Confusing agents with hardware controllers
• Thinking agents only store data
• Assuming agents only display information

2. Which of the following is the correct way to describe an agent's action cycle?

easy

A. Sense environment -> Take action -> Update environment

B. Take action -> Sense environment -> Sleep

C. Sense environment -> Sleep -> Take action

D. Update environment -> Take action -> Sense environment

Solution

1. Step 1: Recall the agent cycle steps

   An agent first senses its environment, then takes an action based on that sensing.

2. Step 2: Match the correct sequence

   Sense environment -> Take action -> Update environment correctly shows sensing first, then acting, then environment update.

3. Final Answer:

   Sense environment -> Take action -> Update environment -> Option A

4. Quick Check:

   Agent cycle = sense then act [OK]

Hint: Agents sense first, then act, then update [OK]

Common Mistakes:

• Mixing order of sensing and acting
• Including sleep incorrectly in cycle
• Ignoring environment update step

3. Consider this simple agent code snippet:

class Agent:
    def __init__(self):
        self.state = 0
    def sense(self, input):
        self.state += input
    def act(self):
        return self.state * 2

agent = Agent()
agent.sense(3)
agent.sense(4)
print(agent.act())

What is the output of this code?

medium

A. 14

B. 7

C. 12

D. 0

Solution

1. Step 1: Calculate state after sensing inputs

   Initial state is 0. After agent.sense(3), state = 3. After agent.sense(4), state = 7.

2. Step 2: Calculate action output

   agent.act() returns state * 2 = 7 * 2 = 14.

3. Final Answer:

   14 -> Option A

4. Quick Check:

   State sum 7 x 2 = 14 [OK]

Hint: Add inputs then multiply by 2 for output [OK]

Common Mistakes:

• Multiplying inputs separately instead of sum
• Using only last input instead of sum
• Confusing state update logic

4. This agent code has a bug:

class Agent:
    def __init__(self):
        self.state = 0
    def sense(self, input):
        self.state = input
    def act(self):
        return self.state * 2

agent = Agent()
agent.sense(3)
agent.sense(4)
print(agent.act())

What is the bug and how to fix it?

medium

A. Bug: sense method missing; Fix: add sense method

B. Bug: act returns wrong value; Fix: return state + 2

C. Bug: state overwritten each sense; Fix: use += to accumulate

D. Bug: state not initialized; Fix: initialize state in act

Solution

1. Step 1: Identify the problem in sense method

   The sense method sets state = input, so previous state is lost on each call.

2. Step 2: Fix by accumulating inputs

   Change state = input to state += input to keep adding inputs.

3. Final Answer:

   Bug: state overwritten each sense; Fix: use += to accumulate -> Option C

4. Quick Check:

   Accumulate inputs with += fixes bug [OK]

Hint: Use += to add inputs, not = to overwrite [OK]

Common Mistakes:

• Thinking act method is wrong
• Adding sense method again unnecessarily
• Initializing state in wrong place

5. You want to design a smart agent that automatically adjusts room temperature based on sensor data. Which approach best fits this task?

hard

A. Use a simple reflex agent that acts only on current sensor reading

B. Use a fixed schedule agent ignoring sensor data

C. Use a random agent that changes temperature randomly

D. Use a model-based agent that keeps track of past temperatures

Solution

1. Step 1: Understand task needs

   Adjusting temperature smartly requires remembering past data to avoid sudden changes.

2. Step 2: Choose agent type

   A model-based agent keeps track of past states, making it suitable for this task.

3. Final Answer:

   Use a model-based agent that keeps track of past temperatures -> Option D

4. Quick Check:

   Smart adjustment needs model-based agent [OK]

Hint: Smart agents remember past data for better decisions [OK]

Common Mistakes:

• Choosing simple reflex agent ignoring history
• Using random or fixed schedule agents
• Not considering past sensor data