Defining success criteria for agents in Agentic AI

Introduction

We set clear goals to know when an agent has done a good job. This helps the agent learn and improve.

When you want a robot to clean a room and know when it's done well.

When you build a chatbot and want to check if it answers questions correctly.

When training a game-playing AI to win or reach a target score.

When creating a recommendation system to see if users like the suggestions.

When developing a self-driving car to ensure it drives safely and reaches destinations.

Syntax

Agentic AI

success_criteria = {
    'goal': 'Description of what success means',
    'metrics': ['list', 'of', 'measurable', 'indicators'],
    'thresholds': {'metric_name': value, ...}
}

The 'goal' explains in simple words what success looks like.

Metrics are numbers or scores that show how well the agent is doing.

Examples

This means the agent must collect all trash and finish within 30 minutes.

Agentic AI

success_criteria = {
    'goal': 'Clean the room with no trash left',
    'metrics': ['trash_collected', 'time_taken'],
    'thresholds': {'trash_collected': 100, 'time_taken': 30}
}

The chatbot should answer 90% correctly and respond within 5 seconds.

Agentic AI

success_criteria = {
    'goal': 'Answer customer questions correctly',
    'metrics': ['accuracy', 'response_time'],
    'thresholds': {'accuracy': 0.9, 'response_time': 5}
}

Sample Model

This code defines what success means for a cleaning robot and checks if the robot met those goals based on its results.

Agentic AI

class Agent:
    def __init__(self, success_criteria):
        self.success_criteria = success_criteria

    def evaluate(self, results):
        for metric, threshold in self.success_criteria['thresholds'].items():
            value = results.get(metric, 0)
            if metric == 'time_taken' or metric == 'response_time':
                if value > threshold:
                    return False
            else:
                if value < threshold:
                    return False
        return True

# Define success criteria for a cleaning robot
success_criteria = {
    'goal': 'Clean the room with no trash left',
    'metrics': ['trash_collected', 'time_taken'],
    'thresholds': {'trash_collected': 100, 'time_taken': 30}
}

# Create agent
agent = Agent(success_criteria)

# Simulated results from agent's task
results = {'trash_collected': 100, 'time_taken': 28}

# Check if agent succeeded
success = agent.evaluate(results)
print(f"Agent success: {success}")

Important Notes

Success criteria should be clear and measurable to guide the agent effectively.

Choose metrics that directly relate to the agent's task and goals.

Thresholds set the minimum level for success; adjust them based on real needs.

Summary

Defining success criteria helps agents know what to achieve.

Use clear goals, measurable metrics, and thresholds to set success.

Evaluating results against criteria shows if the agent succeeded.