Caching and result reuse in Agentic Ai - ML Experiment: Train & Evaluate

Experiment - Caching and result reuse

Problem:You have an AI agent that performs repeated expensive computations on the same inputs during training and inference.

Current Metrics:Computation time per query: 5 seconds; Model throughput: 10 queries per minute.

Issue:The AI agent is slow because it recomputes results for inputs it has already processed, wasting time and resources.

Your Task

Implement caching to store and reuse results of previous computations to reduce computation time by at least 50% without changing model accuracy.

Do not change the core AI model architecture or training data.

Caching must be implemented within the agent's inference pipeline.

Ensure cache consistency and correctness.

Hint 1

Hint 2

Hint 3

Solution

Agentic_ai

import time

class AgentWithCache:
    def __init__(self, model):
        self.model = model
        self.cache = {}

    def compute(self, input_data):
        key = str(input_data)  # Simple key for caching
        if key in self.cache:
            return self.cache[key]
        # Simulate expensive computation
        result = self.model(input_data)
        self.cache[key] = result
        return result

# Dummy model function simulating expensive computation

def dummy_model(x):
    time.sleep(5)  # Simulate delay
    return x * 2

agent = AgentWithCache(dummy_model)

# First call with input 10 (slow)
start = time.time()
output1 = agent.compute(10)
end = time.time()
first_call_time = end - start

# Second call with input 10 (fast, cached)
start = time.time()
output2 = agent.compute(10)
end = time.time()
second_call_time = end - start

print(f"First call output: {output1}, time: {first_call_time:.2f} seconds")
print(f"Second call output: {output2}, time: {second_call_time:.2f} seconds")

Added a cache dictionary to store input-output pairs.

Modified compute method to check cache before running model.

Returned cached result immediately if available to save time.

Results Interpretation

Before caching: Each query took about 5 seconds, limiting throughput to 10 queries per minute.

After caching: Repeated queries took about 0.01 seconds, increasing throughput to 50 queries per minute.

Caching stores results of expensive computations to reuse later, greatly improving speed without affecting accuracy.

Bonus Experiment

Try implementing a cache with a size limit and eviction policy (like Least Recently Used) to manage memory.

💡 Hint

Use collections.OrderedDict or functools.lru_cache decorator to limit cache size and automatically remove old entries.