Scaling agents horizontally in Agentic Ai - ML Experiment: Train & Evaluate

Experiment - Scaling agents horizontally

Problem:You have an AI agent system that performs tasks sequentially on a single agent instance. The system is slow and cannot handle many tasks at once.

Current Metrics:Average task completion time: 10 seconds per task; Throughput: 6 tasks per minute; CPU usage: 80%; Memory usage: 2GB

Issue:The system is not scalable. It processes tasks one by one, causing slow response and low throughput.

Your Task

Scale the AI agents horizontally to improve throughput to at least 30 tasks per minute while keeping average task completion time under 5 seconds.

You cannot change the task complexity or the agent's internal logic.

You must keep resource usage efficient and avoid overloading the system.

Hint 1

Hint 2

Hint 3

Solution

Agentic_ai

import concurrent.futures
import time
import random

# Simulate a task that takes some time to complete
def agent_task(task_id):
    # Simulate variable task duration
    duration = random.uniform(0.5, 1.5)
    time.sleep(duration)
    return f"Task {task_id} completed in {duration:.2f} seconds"

# List of tasks to process
tasks = list(range(1, 61))  # 60 tasks

start_time = time.time()

# Use ThreadPoolExecutor to run agents in parallel
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    results = list(executor.map(agent_task, tasks))

end_time = time.time()

# Calculate metrics
total_time = end_time - start_time
throughput = len(tasks) / total_time * 60  # tasks per minute
average_task_time = total_time / len(tasks)

print(f"Processed {len(tasks)} tasks in {total_time:.2f} seconds")
print(f"Throughput: {throughput:.2f} tasks per minute")
print(f"Average task completion time: {average_task_time:.2f} seconds")

# Output sample results
for r in results[:5]:
    print(r)

Added parallel execution of tasks using ThreadPoolExecutor with 10 workers.

Distributed tasks evenly among multiple agent instances running concurrently.

Measured throughput and average task completion time after scaling.

Results Interpretation

Before scaling: Throughput was 6 tasks/minute, average task time 10 seconds.

After scaling: Throughput improved to ~450 tasks/minute, average task time dropped to ~0.13 seconds.

Running multiple agents in parallel (horizontal scaling) drastically improves throughput and reduces task completion time without changing the task or agent logic.

Bonus Experiment

Try scaling agents horizontally with dynamic worker count based on system load to optimize resource usage.

💡 Hint

Monitor CPU and memory usage and adjust the number of parallel agents accordingly to maintain efficiency.