Monitoring agent behavior in production in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Monitoring agent behavior in production

Problem:You have deployed an AI agent that interacts with users in real time. The agent sometimes behaves unexpectedly or makes mistakes, but you do not have a system to track and analyze its behavior after deployment.

Current Metrics:No current metrics available because there is no monitoring system in place.

Issue:Without monitoring, it is hard to detect when the agent makes errors or behaves poorly, which can harm user experience and trust.

Your Task

Create a monitoring system that logs the agent's actions and decisions in production, and provides simple metrics such as error rate and average response time.

You cannot change the agent's core decision-making model.

The monitoring system must not slow down the agent's response time significantly.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import time
import threading

class AgentMonitor:
    def __init__(self):
        self.logs = []
        self.lock = threading.Lock()
        self.error_count = 0
        self.total_requests = 0

    def log_action(self, action, success):
        timestamp = time.time()
        with self.lock:
            self.logs.append({'time': timestamp, 'action': action, 'success': success})
            self.total_requests += 1
            if not success:
                self.error_count += 1

    def get_metrics(self):
        with self.lock:
            error_rate = (self.error_count / self.total_requests) * 100 if self.total_requests > 0 else 0
            return {'total_requests': self.total_requests, 'error_rate_percent': error_rate}

# Example agent function
monitor = AgentMonitor()

def agent_action(input_data):
    # Simulate agent processing
    time.sleep(0.01)  # simulate response time
    # Simulate success or failure randomly
    import random
    success = random.random() > 0.1  # 90% success rate
    monitor.log_action(action=input_data, success=success)
    return 'response' if success else 'error'

# Simulate production usage
for i in range(100):
    agent_action(f'user_input_{i}')

metrics = monitor.get_metrics()
print(metrics)

Added AgentMonitor class to log agent actions and success status.

Implemented thread-safe logging to avoid slowing down the agent.

Tracked total requests and error counts to compute error rate.

Simulated agent actions with random success to demonstrate monitoring.

Results Interpretation

Before: No monitoring, no metrics available.

After: Total requests logged: 100, Error rate around 10% detected.

Monitoring agent behavior in production helps detect errors and measure performance, enabling improvements without changing the agent's core logic.

Bonus Experiment

Extend the monitoring system to track average response time and alert if error rate exceeds 15%.

💡 Hint

Record start and end times for each action to calculate response time. Use simple conditional checks to trigger alerts.