Experiment - Rate limiting and budget controls

Problem:You have an AI agent that makes API calls to external services. The agent currently makes too many calls, exceeding the allowed rate limit and budget, causing service interruptions and extra costs.

Current Metrics:API calls per minute: 120 (limit is 60), Monthly cost: $300 (budget is $150), Number of failed calls due to rate limit: 30%

Issue:The agent is overusing API calls, causing rate limit errors and exceeding the budget.

Your Task

Implement rate limiting and budget controls to keep API calls under 60 per minute and monthly cost under $150, reducing failed calls to less than 5%.

You cannot reduce the agent's core functionality or accuracy.

You must keep the agent responsive and efficient.

Use only software-based controls (no hardware changes).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Agentic_ai

import time
from collections import deque

class RateLimiter:
    def __init__(self, max_calls_per_minute):
        self.max_calls = max_calls_per_minute
        self.call_times = deque()

    def allow_call(self):
        current_time = time.time()
        while self.call_times and current_time - self.call_times[0] >= 60:
            self.call_times.popleft()
        if len(self.call_times) < self.max_calls:
            self.call_times.append(current_time)
            return True
        return False

class BudgetController:
    def __init__(self, max_budget):
        self.max_budget = max_budget
        self.current_cost = 0

    def can_spend(self, cost):
        return (self.current_cost + cost) <= self.max_budget

    def spend(self, cost):
        self.current_cost += cost

class Agent:
    def __init__(self, rate_limiter, budget_controller, cost_per_call=0.05):
        self.rate_limiter = rate_limiter
        self.budget_controller = budget_controller
        self.cost_per_call = cost_per_call

    def make_api_call(self):
        if not self.rate_limiter.allow_call():
            print("Rate limit exceeded, delaying call.")
            return False
        if not self.budget_controller.can_spend(self.cost_per_call):
            print("Budget exceeded, stopping calls.")
            return False
        # Simulate API call
        time.sleep(0.01)  # simulate network delay
        self.budget_controller.spend(self.cost_per_call)
        print("API call successful.")
        return True

# Setup
rate_limiter = RateLimiter(max_calls_per_minute=60)
budget_controller = BudgetController(max_budget=150)
agent = Agent(rate_limiter, budget_controller)

# Simulate calls
successful_calls = 0
failed_calls = 0
for _ in range(2000):
    if agent.make_api_call():
        successful_calls += 1
    else:
        failed_calls += 1
    time.sleep(1)  # simulate time between calls

print(f"Successful calls: {successful_calls}")
print(f"Failed calls: {failed_calls}")
print(f"Total cost: ${budget_controller.current_cost:.2f}")

Added a RateLimiter class to limit API calls to 60 per minute using a sliding window.

Added a BudgetController class to track and limit total spending to $150.

Modified the agent to check rate limits and budget before making calls.

Added delays and stopping conditions to prevent exceeding limits.

Results Interpretation

Before: 120 calls/min, $300 cost, 30% failed calls due to rate limit.

After: <=60 calls/min, <=$150 cost, <5% failed calls.

Implementing rate limiting and budget controls helps prevent overuse of resources, reduces errors, and keeps costs manageable without sacrificing core functionality.

Bonus Experiment

Try implementing a caching mechanism to store frequent API responses and reduce the number of calls needed.

💡 Hint

Use a dictionary with expiration times to store and reuse responses for repeated requests.