Agent perception-reasoning-action loop in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Agent perception-reasoning-action loop

Which metric matters for Agent perception-reasoning-action loop and WHY

The agent perception-reasoning-action loop involves sensing the environment, making decisions, and acting. To evaluate this loop, accuracy of perception (correctly understanding inputs) and decision quality (correct actions chosen) are key. Metrics like precision and recall help measure how well the agent detects important events (perception). For reasoning and action, success rate or reward from actions shows if decisions lead to good outcomes. These metrics matter because a wrong perception or poor reasoning leads to wrong actions, reducing overall agent effectiveness.

Confusion matrix example for perception accuracy

    | Predicted Positive | Predicted Negative |
    |--------------------|--------------------|
    | True Positive (TP)  | False Positive (FP) |
    | False Negative (FN) | True Negative (TN)  |

    Example:
    TP = 80 (correctly detected events)
    FP = 20 (false alarms)
    FN = 10 (missed events)
    TN = 90 (correctly ignored)

    Total samples = 80 + 20 + 10 + 90 = 200
    

From this, precision = 80 / (80 + 20) = 0.8, recall = 80 / (80 + 10) = 0.89

Precision vs Recall tradeoff in agent perception

If the agent is too cautious, it may miss important events (low recall), causing bad decisions. If it is too sensitive, it may raise many false alarms (low precision), wasting resources on unnecessary actions.

For example, a security robot must detect intruders. High recall means it catches most intruders but may trigger false alarms (low precision). High precision means fewer false alarms but might miss some intruders (low recall). Balancing these depends on what is worse: missing threats or false alerts.

Good vs Bad metric values for agent loops

Good: Precision and recall above 0.85, indicating reliable perception. High success rate or reward from actions, showing effective reasoning and acting.

Bad: Precision or recall below 0.5 means poor perception, leading to wrong decisions. Low success rate means actions do not achieve goals, possibly due to bad reasoning.

Common pitfalls in evaluating agent loops

• Accuracy paradox: High overall accuracy can hide poor detection of rare but important events.
• Data leakage: Using future information in training can inflate metrics unrealistically.
• Overfitting: Agent performs well on training scenarios but fails in new environments.
• Ignoring action outcomes: Good perception but poor action evaluation misses the full loop quality.

Self-check question

Your agent has 98% accuracy in perception but only 12% recall on detecting critical events. Is it good for production? Why not?

Answer: No, it is not good. Although accuracy is high, the agent misses 88% of critical events (low recall). This means it often fails to detect important situations, leading to poor decisions and actions. High recall is crucial for safety and effectiveness.

Key Result

Precision and recall are key to measure perception quality; success rate measures reasoning-action effectiveness.

Practice

1. What is the correct order of steps in the agent perception-reasoning-action loop?

easy

A. Act, Reason, Perceive

B. Act, Perceive, Reason

C. Reason, Act, Perceive

D. Perceive, Reason, Act

Solution

1. Step 1: Understand the agent loop components

   The agent loop consists of three main steps: perceiving the environment, reasoning about the information, and then acting based on that reasoning.

2. Step 2: Identify the correct sequence

   The agent must first perceive to gather data, then reason to decide what to do, and finally act to affect the environment.

3. Final Answer:

   Perceive, Reason, Act -> Option D

4. Quick Check:

   Agent loop order = Perceive, Reason, Act [OK]

Hint: Remember: see first, think second, do last [OK]

Common Mistakes:

• Mixing up the order of reasoning and acting
• Thinking action happens before perception
• Skipping the reasoning step

2. Which of the following code snippets correctly represents the agent loop structure in Python?

easy

A. while True:
reason()
act()
perceive()

B. while True:
act()
perceive()
reason()

C. while True:
perceive()
reason()
act()

D. while True:
act()
reason()
perceive()

Solution

1. Step 1: Check the order of function calls

   The agent loop must call perceive() first, then reason(), then act() inside the loop.

2. Step 2: Verify the code snippet matches this order

   while True: perceive() reason() act() calls perceive(), then reason(), then act(), which matches the correct loop order.

3. Final Answer:

   while True:\n perceive()\n reason()\n act() -> Option C

4. Quick Check:

   Code order = perceive, reason, act [OK]

Hint: Loop order matches perception, reasoning, then action [OK]

Common Mistakes:

• Calling act() before perceive()
• Swapping reason() and act() calls
• Incorrect indentation causing syntax errors

3. Given this simplified agent loop code, what will be printed?

def perceive():
    return "data"
def reason(data):
    return data.upper()
def act(result):
    print(f"Action: {result}")

for _ in range(2):
    data = perceive()
    result = reason(data)
    act(result)

medium

A. Action: DATA\nAction: DATA

B. Error: missing argument in reason()

C. Action: Data\nAction: Data

D. Action: data\nAction: data

Solution

1. Step 1: Trace the function calls in the loop

   Each loop iteration calls perceive() returning "data", then reason(data) converts it to uppercase "DATA", then act(result) prints "Action: DATA".

2. Step 2: Repeat for two iterations

   The loop runs twice, so the print happens twice with "Action: DATA" each time.

3. Final Answer:

   Action: DATA\nAction: DATA -> Option A

4. Quick Check:

   Uppercase output printed twice = Action: DATA [OK]

Hint: Check function returns and loop count carefully [OK]

Common Mistakes:

• Assuming reason() returns original lowercase
• Forgetting to pass argument to reason()
• Confusing print output formatting

4. Identify the error in this agent loop code snippet:

def perceive():
    return "info"
def reason():
    # missing parameter
    return "processed"
def act(result):
    print(result)

while True:
    data = perceive()
    result = reason()
    act(result)
    break

medium

A. act() should not print the result

B. reason() should accept an argument but does not

C. perceive() should not return a value

D. while loop should not have a break

Solution

1. Step 1: Check function parameters and calls

   perceive() returns "info" which is stored in data, but reason() is called without arguments though it should process data.

2. Step 2: Identify mismatch causing error

   reason() lacks a parameter to receive data, so calling reason() without argument causes a logic error or mismatch.

3. Final Answer:

   reason() should accept an argument but does not -> Option B

4. Quick Check:

   Function parameter mismatch = reason() missing argument [OK]

Hint: Match function parameters with calls exactly [OK]

Common Mistakes:

• Ignoring missing parameter in reason()
• Thinking perceive() should not return data
• Assuming break is incorrect in loop

5. You want to design an agent that perceives temperature, reasons if it's too hot or cold, and acts by turning on a heater or cooler. Which code snippet correctly implements this agent loop?

hard

A. def perceive():
return 30

def reason(temp):
if temp > 25:
return "cooler"
elif temp < 18:
return "heater"
else:
return "off"

def act(action):
print(f"Turn {action} on")

while True:
temp = perceive()
action = reason(temp)
act(action)
break

B. def perceive():
return 30

def reason():
if temp > 25:
return "cooler"
elif temp < 18:
return "heater"
else:
return "off"

def act(action):
print(f"Turn {action} on")

while True:
temp = perceive()
action = reason()
act(action)
break

C. def perceive():
return 30

def reason(temp):
if temp < 18:
return "cooler"
elif temp > 25:
return "heater"
else:
return "off"

def act(action):
print(f"Turn {action} on")

while True:
temp = perceive()
action = reason(temp)
act(action)
break

D. def perceive():
return 30

def reason(temp):
if temp > 25:
return "heater"
elif temp < 18:
return "cooler"
else:
return "off"

def act(action):
print(f"Turn {action} on")

while True:
temp = perceive()
action = reason(temp)
act(action)
break

Solution

1. Step 1: Check perception and reasoning logic

   perceive() returns temperature 30. reason(temp) correctly returns "cooler" if temp > 25, "heater" if temp < 18, else "off".

2. Step 2: Verify action and loop structure

   act(action) prints the correct command. The loop calls perceive(), reason(temp), and act(action) in correct order and breaks after one iteration.

3. Final Answer:

   Option A correctly implements the agent loop with proper logic and function calls -> Option A

4. Quick Check:

   Correct logic and loop = def perceive(): return 30 def reason(temp): if temp > 25: return "cooler" elif temp < 18: return "heater" else: return "off" def act(action): print(f"Turn {action} on") while True: temp = perceive() action = reason(temp) act(action) break [OK]

Hint: Match temperature conditions with correct actions [OK]

Common Mistakes:

• Missing parameter in reason() function
• Swapping heater and cooler logic
• Calling reason() without argument