Scaling agents horizontally in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Scaling agents horizontally

Problem:You have an AI agent system that performs tasks sequentially on a single agent instance. The system is slow and cannot handle many tasks at once.

Current Metrics:Average task completion time: 10 seconds per task; Throughput: 6 tasks per minute; CPU usage: 80%; Memory usage: 2GB

Issue:The system is not scalable. It processes tasks one by one, causing slow response and low throughput.

Your Task

Scale the AI agents horizontally to improve throughput to at least 30 tasks per minute while keeping average task completion time under 5 seconds.

You cannot change the task complexity or the agent's internal logic.

You must keep resource usage efficient and avoid overloading the system.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import concurrent.futures
import time
import random

# Simulate a task that takes some time to complete
def agent_task(task_id):
    # Simulate variable task duration
    duration = random.uniform(0.5, 1.5)
    time.sleep(duration)
    return f"Task {task_id} completed in {duration:.2f} seconds"

# List of tasks to process
tasks = list(range(1, 61))  # 60 tasks

start_time = time.time()

# Use ThreadPoolExecutor to run agents in parallel
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    results = list(executor.map(agent_task, tasks))

end_time = time.time()

# Calculate metrics
total_time = end_time - start_time
throughput = len(tasks) / total_time * 60  # tasks per minute
average_task_time = total_time / len(tasks)

print(f"Processed {len(tasks)} tasks in {total_time:.2f} seconds")
print(f"Throughput: {throughput:.2f} tasks per minute")
print(f"Average task completion time: {average_task_time:.2f} seconds")

# Output sample results
for r in results[:5]:
    print(r)

Added parallel execution of tasks using ThreadPoolExecutor with 10 workers.

Distributed tasks evenly among multiple agent instances running concurrently.

Measured throughput and average task completion time after scaling.

Results Interpretation

Before scaling: Throughput was 6 tasks/minute, average task time 10 seconds.

After scaling: Throughput improved to ~450 tasks/minute, average task time dropped to ~0.13 seconds.

Running multiple agents in parallel (horizontal scaling) drastically improves throughput and reduces task completion time without changing the task or agent logic.

Bonus Experiment

Try scaling agents horizontally with dynamic worker count based on system load to optimize resource usage.

💡 Hint

Monitor CPU and memory usage and adjust the number of parallel agents accordingly to maintain efficiency.

Practice

1. What does scaling agents horizontally mean in agentic AI?

easy

A. Adding more agents to share and run tasks in parallel

B. Making one agent work faster by improving its code

C. Reducing the number of agents to save resources

D. Changing the task to fit a single agent's ability

Solution

1. Step 1: Understand the term 'scaling horizontally'

   Scaling horizontally means increasing the number of units (agents) to handle more work simultaneously.

2. Step 2: Apply to agentic AI context

   In agentic AI, this means adding more agents to share tasks and run them in parallel, speeding up processing.

3. Final Answer:

   Adding more agents to share and run tasks in parallel -> Option A

4. Quick Check:

   Scaling horizontally = Adding more agents [OK]

Hint: More agents working together means horizontal scaling [OK]

Common Mistakes:

• Confusing horizontal scaling with making one agent faster
• Thinking scaling means reducing agents
• Assuming scaling changes the task itself

2. Which of the following is the correct way to start multiple agents in parallel in Python?

easy

A. for agent in agents:
agent.start()

B. for agent in agents:
agent.run()

C. for agent in agents:
agent.execute()

D. for agent in agents:
agent.parallel()

Solution

1. Step 1: Identify the method to start agents in parallel

   In many agent frameworks, start() is used to begin an agent's process or thread asynchronously.

2. Step 2: Compare options

   run() usually runs synchronously blocking the loop, execute() and parallel() are not standard methods.

3. Final Answer:

   for agent in agents: agent.start() -> Option A

4. Quick Check:

   Use start() to launch agents in parallel [OK]

Hint: Use start() to run agents asynchronously [OK]

Common Mistakes:

• Using run() which blocks instead of start()
• Assuming execute() or parallel() are valid methods
• Not looping over all agents

3. Given this code snippet for scaling agents horizontally, what will be the output?

class Agent:
    def __init__(self, id):
        self.id = id
    def run(self):
        print(f"Agent {self.id} running")

agents = [Agent(i) for i in range(3)]
for agent in agents:
    agent.run()

medium

A. Agent 3 running

B. Agent running\nAgent running\nAgent running

C. No output, code has error

D. Agent 0 running\nAgent 1 running\nAgent 2 running

Solution

1. Step 1: Understand the Agent class and its run method

   The run method prints the agent's id with the message "Agent {id} running".

2. Step 2: Analyze the loop over agents

   There are 3 agents with ids 0, 1, 2. The loop calls run() on each, printing their messages in order.

3. Final Answer:

   Agent 0 running Agent 1 running Agent 2 running -> Option D

4. Quick Check:

   Each agent prints its id running [OK]

Hint: Each agent prints its id in order [OK]

Common Mistakes:

• Thinking all agents print the same message without id
• Assuming only one agent runs
• Believing code has syntax error

4. This code tries to scale agents horizontally but does not run agents in parallel. What is the error?

class Agent:
    def run(self):
        print("Running")

agents = [Agent() for _ in range(3)]
for agent in agents:
    agent.run()

medium

A. The list comprehension syntax is wrong

B. Agent class is missing an __init__ method

C. Agents are run sequentially, not in parallel

D. The run method should be named start

Solution

1. Step 1: Check how agents are executed

   The for loop calls run() on each agent one after another, so execution is sequential.

2. Step 2: Understand parallel execution requirement

   To scale horizontally, agents must run in parallel, e.g., using threads or async calls, not sequential calls.

3. Final Answer:

   Agents are run sequentially, not in parallel -> Option C

4. Quick Check:

   Sequential run ≠ horizontal scaling [OK]

Hint: Sequential calls don't scale horizontally [OK]

Common Mistakes:

• Thinking missing __init__ causes no parallelism
• Believing list comprehension is incorrect
• Assuming run must be renamed to start

5. You want to scale 5 agents horizontally to process independent tasks faster. Which approach best achieves this in Python?

hard

A. Run all agents sequentially in a single loop

B. Run each agent's task in a separate thread using threading.Thread

C. Use a single agent to process all tasks one by one

D. Run agents in a loop but wait for each to finish before starting next

Solution

1. Step 1: Understand the goal of horizontal scaling

   We want to run multiple agents at the same time to speed up processing independent tasks.

2. Step 2: Evaluate options for parallel execution

   Using threading.Thread runs agents concurrently, achieving horizontal scaling. Sequential loops or waiting block parallelism.

3. Final Answer:

   Run each agent's task in a separate thread using threading.Thread -> Option B

4. Quick Check:

   Threads enable parallel agent execution [OK]

Hint: Use threads to run agents in parallel [OK]

Common Mistakes:

• Running agents sequentially thinking it's parallel
• Using one agent for all tasks ignoring scaling
• Starting agents but waiting for each to finish before next