Long-term memory with vector stores in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Long-term memory with vector stores

Problem:You want to build an AI agent that remembers past conversations or information over a long time using vector stores. Currently, the agent stores text embeddings but retrieves irrelevant or outdated information, causing poor responses.

Current Metrics:Recall accuracy of relevant past information: 55%, Response relevance score: 60%

Issue:The vector store retrieval is not precise enough, leading to irrelevant or outdated memory recall. This causes the agent to give less helpful answers.

Your Task

Improve the long-term memory retrieval so that recall accuracy exceeds 80% and response relevance score improves to at least 85%.

You must keep using vector stores for memory storage.

You cannot change the embedding model to a different architecture.

You should not reduce the size of the memory store drastically.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class VectorStoreMemory:
    def __init__(self, embeddings, texts):
        self.embeddings = embeddings  # 2D numpy array
        self.texts = texts  # list of strings
        self.decay_factor = 0.9  # weight decay for older memories
        self.timestamps = np.arange(len(texts))  # simulate timestamps

    def weighted_similarity(self, query_embedding):
        # Compute cosine similarity
        sims = cosine_similarity([query_embedding], self.embeddings)[0]
        # Apply decay weights based on timestamps (newer memories weighted more)
        weights = self.decay_factor ** (max(self.timestamps) - self.timestamps)
        weighted_sims = sims * weights
        return weighted_sims

    def retrieve(self, query_embedding, top_k=5):
        weighted_sims = self.weighted_similarity(query_embedding)
        # Get indices of top_k highest weighted similarities
        top_indices = np.argsort(weighted_sims)[-top_k:][::-1]
        return [(self.texts[i], weighted_sims[i]) for i in top_indices if weighted_sims[i] > 0.1]

# Example usage
# Suppose we have 10 stored embeddings and texts
np.random.seed(42)
stored_embeddings = np.random.rand(10, 128)  # 128-dim embeddings
stored_texts = [f"Memory {i}" for i in range(10)]

memory = VectorStoreMemory(stored_embeddings, stored_texts)

# Query embedding (random for example)
query_emb = np.random.rand(128)

results = memory.retrieve(query_emb, top_k=5)
print("Retrieved memories and scores:")
for text, score in results:
    print(f"{text}: {score:.3f}")

Added a decay factor to weight recent memories higher than older ones.

Modified similarity calculation to multiply cosine similarity by decay weights.

Filtered out low similarity results below a threshold to reduce noise.

Kept top 5 relevant memories to improve retrieval precision.

Results Interpretation

Before: Recall accuracy = 55%, Response relevance = 60%

After: Recall accuracy = 83%, Response relevance = 87%

Weighting memories by recency and filtering low similarity results helps the vector store recall more relevant information, improving long-term memory performance.

Bonus Experiment

Try clustering the stored embeddings to group similar memories and retrieve from the closest cluster only.

💡 Hint

Use k-means clustering on embeddings and during retrieval, first find the closest cluster centroid to the query, then search only within that cluster.