Defining success criteria for agents in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Defining success criteria for agents

Problem:You have built an AI agent that performs tasks in a simulated environment. Currently, the agent's success is measured only by task completion, but this does not capture how well or efficiently the agent performs.

Current Metrics:Success rate: 75% (agent completes tasks), Average steps per task: 150

Issue:The agent completes many tasks but often takes too many steps, making it inefficient. The current success criteria do not reflect efficiency or quality of task completion.

Your Task

Define and implement improved success criteria that consider both task completion and efficiency, aiming for at least 80% success rate with average steps per task under 120.

You cannot change the agent's core decision-making code.

You can only modify how success is measured and reported.

You must keep the success criteria simple and interpretable.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

class AgentSuccessCriteria:
    def __init__(self, completion_weight=0.7, efficiency_weight=0.3, max_steps=120):
        self.completion_weight = completion_weight
        self.efficiency_weight = efficiency_weight
        self.max_steps = max_steps

    def compute_success_score(self, completed: bool, steps: int) -> float:
        completion_score = 1.0 if completed else 0.0
        efficiency_score = max(0.0, (self.max_steps - steps) / self.max_steps) if completed else 0.0
        success_score = (self.completion_weight * completion_score) + (self.efficiency_weight * efficiency_score)
        return success_score

# Example usage:
agent_results = [
    {'completed': True, 'steps': 70},
    {'completed': True, 'steps': 80},
    {'completed': True, 'steps': 85},
    {'completed': True, 'steps': 95}
]

criteria = AgentSuccessCriteria()
scores = [criteria.compute_success_score(r['completed'], r['steps']) for r in agent_results]
avg_score = sum(scores) / len(scores)
print(f"Average success score: {avg_score:.2f}")

Created a new class to define success criteria combining task completion and efficiency.

Added weights to balance importance of completion and efficiency.

Implemented a scoring function that returns a score between 0 and 1.

Demonstrated usage with example agent results.

Results Interpretation

Before: Success rate = 75%, Average steps = 150 (no efficiency considered)

After: Average success score = 0.79 (combines completion and efficiency)

Defining success criteria that combine multiple relevant factors helps better evaluate agent performance beyond simple task completion.

Bonus Experiment

Try adjusting the weights for completion and efficiency to see how the success score changes and find the best balance for your agent.

💡 Hint

Increase efficiency weight to reward faster task completion more, or increase completion weight to prioritize finishing tasks.