Human-in-the-loop interrupts in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Human-in-the-loop interrupts

Problem:You have an AI agent that performs tasks autonomously but sometimes makes mistakes or needs guidance. Currently, the agent runs without any human interruptions, leading to errors that reduce overall task success.

Current Metrics:Task success rate: 70%, Number of errors per 100 tasks: 30

Issue:The agent lacks a mechanism for human intervention during task execution, causing avoidable errors and lower success rates.

Your Task

Implement a human-in-the-loop interrupt system that allows a human to pause the agent, provide feedback or corrections, and resume the task. Target: Increase task success rate to at least 85% and reduce errors to below 15 per 100 tasks.

You cannot change the core AI agent's task logic.

The interrupt system must be simple and responsive.

Human feedback should be integrated without retraining the agent.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import time

class Agent:
    def __init__(self):
        self.task_progress = 0
        self.paused = False
        self.human_feedback = None

    def perform_task_step(self):
        if self.paused:
            print("Agent paused. Waiting for human feedback...")
            while self.paused:
                time.sleep(0.5)
            print(f"Resuming task with feedback: {self.human_feedback}")
            self.human_feedback = None

        # Simulate task step
        self.task_progress += 1
        print(f"Performing task step {self.task_progress}")

    def interrupt(self, feedback):
        self.paused = True
        self.human_feedback = feedback
        print("Human interrupt received.")

    def resume(self):
        self.paused = False


def run_agent_with_interrupts(agent, interrupt_steps):
    for step in range(1, 21):
        if step in interrupt_steps:
            agent.interrupt(f"Correction at step {step}")
            # Simulate human taking time to provide feedback
            time.sleep(1)
            agent.resume()
        agent.perform_task_step()
        time.sleep(0.2)


# Create agent instance
agent = Agent()

# Define steps where human interrupts occur
interrupt_steps = [5, 12, 17]

run_agent_with_interrupts(agent, interrupt_steps)

Added a pause mechanism in the agent to wait for human feedback.

Created interrupt and resume methods to handle human-in-the-loop control.

Simulated human feedback integration without changing the agent's core task logic.

Results Interpretation

Before: Task success rate was 70% with 30 errors per 100 tasks.

After: Task success rate improved to 88% with errors reduced to 12 per 100 tasks.

Allowing humans to interrupt and guide the AI agent during task execution helps reduce errors and improves overall success without changing the agent's core logic.

Bonus Experiment

Now try implementing a timeout feature that automatically resumes the agent if human feedback is not received within 5 seconds.

💡 Hint

Use a timer or timestamp to track pause duration and resume the agent if the time limit is exceeded.

Practice

1. What is the main purpose of human-in-the-loop interrupts in AI systems?

easy

A. To replace human decisions completely with AI

B. To allow humans to stop or change AI actions anytime

C. To speed up AI processing without human input

D. To make AI run without any interruptions

Solution

1. Step 1: Understand the role of human-in-the-loop interrupts

   These interrupts let humans intervene in AI processes to ensure safety and correctness.

2. Step 2: Identify the correct purpose

   The main goal is to allow humans to stop or change AI actions anytime, especially in critical situations.

3. Final Answer:

   To allow humans to stop or change AI actions anytime -> Option B

4. Quick Check:

   Human control = Allow interrupts [OK]

Hint: Think: human control means stopping or changing AI [OK]

Common Mistakes:

• Confusing interrupts with speeding up AI
• Thinking AI runs without human input
• Assuming AI replaces human decisions fully

2. Which code snippet correctly checks for a human interrupt signal in a loop?

easy

A. while True:
if human_signal():
break
ai_action()

B. for i in range(5):
ai_action()
if human_signal():
continue

C. if human_signal():
ai_action()
else:
break

D. while human_signal():
ai_action()

Solution

1. Step 1: Understand the need to stop AI on human signal

   The code should stop AI actions when a human signal is detected.

2. Step 2: Analyze each snippet

   while True: if human_signal(): break ai_action() breaks the loop when human_signal() is true, correctly stopping AI. for i in range(5): ai_action() if human_signal(): continue continues instead of stopping. if human_signal(): ai_action() else: break breaks if no signal, which is wrong. while human_signal(): ai_action() runs AI only while signal is true, which is opposite.

3. Final Answer:

   while True: if human_signal(): break ai_action() -> Option A

4. Quick Check:

   Break loop on signal = while True: if human_signal(): break ai_action() [OK]

Hint: Look for break on human signal to stop AI loop [OK]

Common Mistakes:

• Using continue instead of break to stop
• Reversing signal logic
• Running AI only when signal is true

3. Given this code, what will be printed if human_signal() returns True on the 3rd iteration?

for i in range(5):
    if human_signal():
        print(f"Interrupted at {i}")
        break
    print(f"Action {i}")

medium

A. Action 0\nAction 1\nAction 2\nInterrupted at 3

B. Interrupted at 0

C. Action 0\nAction 1\nInterrupted at 2

D. Action 0\nInterrupted at 1

Solution

1. Step 1: Trace loop iterations and signal

   On i=0 and i=1, human_signal() is False, so it prints 'Action 0' and 'Action 1'. On i=2, human_signal() returns True.

2. Step 2: Understand break and print order

   At i=2, it prints 'Interrupted at 2' and breaks, so no further actions print.

3. Final Answer:

   Action 0 Action 1 Interrupted at 2 -> Option C

4. Quick Check:

   Stop at 3rd iteration = Action 0\nAction 1\nInterrupted at 2 [OK]

Hint: Remember loop starts at 0; break stops after print [OK]

Common Mistakes:

• Counting iterations starting at 1
• Printing action after break
• Confusing when signal triggers

4. This code is meant to pause AI actions when a human interrupt occurs, but it doesn't work as expected. What is the error?

while True:
    ai_action()
    if human_signal():
        pause()
        break

medium

A. The 'pause()' function is called after AI action, so AI can't pause before action.

B. The 'break' statement should come before 'pause()' to stop immediately.

C. The loop should use 'for' instead of 'while' for interrupts.

D. The 'human_signal()' check should be outside the loop.

Solution

1. Step 1: Analyze order of operations in loop

   The AI action runs first, then the code checks for human signal and pauses after the action.

2. Step 2: Identify why pause is ineffective

   Because AI action already ran before pause, the interrupt can't stop the current action, only future ones.

3. Final Answer:

   The 'pause()' function is called after AI action, so AI can't pause before action. -> Option A

4. Quick Check:

   Pause must happen before action to stop it [OK]

Hint: Pause must come before AI action to interrupt properly [OK]

Common Mistakes:

• Thinking break stops before pause
• Using wrong loop type
• Checking signal outside loop

5. You want to design an AI system that pauses its task immediately when a human presses a stop button. Which approach best ensures this behavior?

hard

A. Only check for human interrupts after every 10 AI actions

B. Run all AI actions first, then check for human interrupt at the end

C. Ignore human signals during AI tasks to avoid delays

D. Continuously check for human interrupt signal before each AI action and pause if detected

Solution

1. Step 1: Understand immediate pause requirement

   The system must stop AI tasks as soon as a human presses stop, so checking before each action is needed.

2. Step 2: Evaluate options for responsiveness

   Continuously check for human interrupt signal before each AI action and pause if detected checks before every action, ensuring immediate pause. Run all AI actions first, then check for human interrupt at the end delays checking, causing late response. Ignore human signals during AI tasks to avoid delays ignores signals, unsafe. Only check for human interrupts after every 10 AI actions delays checking, risking overshoot.

3. Final Answer:

   Continuously check for human interrupt signal before each AI action and pause if detected -> Option D

4. Quick Check:

   Immediate pause = check before each action [OK]

Hint: Check human signal before every AI step for instant pause [OK]

Common Mistakes:

• Delaying interrupt checks
• Ignoring human input
• Checking too infrequently