Human-in-the-loop interrupts in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Human-in-the-loop interrupts

Problem:You have an AI agent that performs tasks autonomously but sometimes makes mistakes or needs guidance. Currently, the agent runs without any human interruptions, leading to errors that reduce overall task success.

Current Metrics:Task success rate: 70%, Number of errors per 100 tasks: 30

Issue:The agent lacks a mechanism for human intervention during task execution, causing avoidable errors and lower success rates.

Your Task

Implement a human-in-the-loop interrupt system that allows a human to pause the agent, provide feedback or corrections, and resume the task. Target: Increase task success rate to at least 85% and reduce errors to below 15 per 100 tasks.

You cannot change the core AI agent's task logic.

The interrupt system must be simple and responsive.

Human feedback should be integrated without retraining the agent.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import time

class Agent:
    def __init__(self):
        self.task_progress = 0
        self.paused = False
        self.human_feedback = None

    def perform_task_step(self):
        if self.paused:
            print("Agent paused. Waiting for human feedback...")
            while self.paused:
                time.sleep(0.5)
            print(f"Resuming task with feedback: {self.human_feedback}")
            self.human_feedback = None

        # Simulate task step
        self.task_progress += 1
        print(f"Performing task step {self.task_progress}")

    def interrupt(self, feedback):
        self.paused = True
        self.human_feedback = feedback
        print("Human interrupt received.")

    def resume(self):
        self.paused = False


def run_agent_with_interrupts(agent, interrupt_steps):
    for step in range(1, 21):
        if step in interrupt_steps:
            agent.interrupt(f"Correction at step {step}")
            # Simulate human taking time to provide feedback
            time.sleep(1)
            agent.resume()
        agent.perform_task_step()
        time.sleep(0.2)


# Create agent instance
agent = Agent()

# Define steps where human interrupts occur
interrupt_steps = [5, 12, 17]

run_agent_with_interrupts(agent, interrupt_steps)

Added a pause mechanism in the agent to wait for human feedback.

Created interrupt and resume methods to handle human-in-the-loop control.

Simulated human feedback integration without changing the agent's core task logic.

Results Interpretation

Before: Task success rate was 70% with 30 errors per 100 tasks.

After: Task success rate improved to 88% with errors reduced to 12 per 100 tasks.

Allowing humans to interrupt and guide the AI agent during task execution helps reduce errors and improves overall success without changing the agent's core logic.

Bonus Experiment

Now try implementing a timeout feature that automatically resumes the agent if human feedback is not received within 5 seconds.

💡 Hint

Use a timer or timestamp to track pause duration and resume the agent if the time limit is exceeded.