Long-term memory with vector stores in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Long-term memory with vector stores

Problem:You want to build an AI agent that remembers past conversations or information over a long time using vector stores. Currently, the agent stores text embeddings but retrieves irrelevant or outdated information, causing poor responses.

Current Metrics:Recall accuracy of relevant past information: 55%, Response relevance score: 60%

Issue:The vector store retrieval is not precise enough, leading to irrelevant or outdated memory recall. This causes the agent to give less helpful answers.

Your Task

Improve the long-term memory retrieval so that recall accuracy exceeds 80% and response relevance score improves to at least 85%.

You must keep using vector stores for memory storage.

You cannot change the embedding model to a different architecture.

You should not reduce the size of the memory store drastically.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class VectorStoreMemory:
    def __init__(self, embeddings, texts):
        self.embeddings = embeddings  # 2D numpy array
        self.texts = texts  # list of strings
        self.decay_factor = 0.9  # weight decay for older memories
        self.timestamps = np.arange(len(texts))  # simulate timestamps

    def weighted_similarity(self, query_embedding):
        # Compute cosine similarity
        sims = cosine_similarity([query_embedding], self.embeddings)[0]
        # Apply decay weights based on timestamps (newer memories weighted more)
        weights = self.decay_factor ** (max(self.timestamps) - self.timestamps)
        weighted_sims = sims * weights
        return weighted_sims

    def retrieve(self, query_embedding, top_k=5):
        weighted_sims = self.weighted_similarity(query_embedding)
        # Get indices of top_k highest weighted similarities
        top_indices = np.argsort(weighted_sims)[-top_k:][::-1]
        return [(self.texts[i], weighted_sims[i]) for i in top_indices if weighted_sims[i] > 0.1]

# Example usage
# Suppose we have 10 stored embeddings and texts
np.random.seed(42)
stored_embeddings = np.random.rand(10, 128)  # 128-dim embeddings
stored_texts = [f"Memory {i}" for i in range(10)]

memory = VectorStoreMemory(stored_embeddings, stored_texts)

# Query embedding (random for example)
query_emb = np.random.rand(128)

results = memory.retrieve(query_emb, top_k=5)
print("Retrieved memories and scores:")
for text, score in results:
    print(f"{text}: {score:.3f}")

Added a decay factor to weight recent memories higher than older ones.

Modified similarity calculation to multiply cosine similarity by decay weights.

Filtered out low similarity results below a threshold to reduce noise.

Kept top 5 relevant memories to improve retrieval precision.

Results Interpretation

Before: Recall accuracy = 55%, Response relevance = 60%

After: Recall accuracy = 83%, Response relevance = 87%

Weighting memories by recency and filtering low similarity results helps the vector store recall more relevant information, improving long-term memory performance.

Bonus Experiment

Try clustering the stored embeddings to group similar memories and retrieve from the closest cluster only.

💡 Hint

Use k-means clustering on embeddings and during retrieval, first find the closest cluster centroid to the query, then search only within that cluster.

Practice

1. What is the main purpose of using a vector store in long-term memory for AI agents?

easy

A. To replace all databases with text files

B. To store images and videos directly

C. To slow down data retrieval for security

D. To save information as lists of numbers for quick searching

Solution

1. Step 1: Understand vector store role

   Vector stores save data as number lists (vectors) to represent information compactly.

2. Step 2: Identify purpose in AI memory

   This allows fast searching by comparing vector similarity, making recall efficient.

3. Final Answer:

   To save information as lists of numbers for quick searching -> Option D

4. Quick Check:

   Vector store = number lists for fast recall [OK]

Hint: Vector stores = numbers for fast search [OK]

Common Mistakes:

• Thinking vector stores save raw images or videos
• Confusing vector stores with simple text files
• Assuming vector stores slow down retrieval

2. Which of the following is the correct way to add a vector to a vector store in Python-like pseudocode?

easy

A. vector_store.insert('doc1', [0.1, 0.2, 0.3])

B. vector_store.push_vector([0.1, 0.2, 0.3], 'doc1')

C. vector_store.add_vector(id='doc1', vector=[0.1, 0.2, 0.3])

D. vector_store.append_vector('doc1', vector=[0.1, 0.2, 0.3])

Solution

1. Step 1: Identify typical vector store method

   Common vector stores use an add_vector method with id and vector parameters.

2. Step 2: Match correct syntax

   vector_store.add_vector(id='doc1', vector=[0.1, 0.2, 0.3]) matches this pattern: add_vector(id='doc1', vector=[0.1, 0.2, 0.3]). Others use incorrect method names or argument order.

3. Final Answer:

   vector_store.add_vector(id='doc1', vector=[0.1, 0.2, 0.3]) -> Option C

4. Quick Check:

   Correct method and argument names = vector_store.add_vector(id='doc1', vector=[0.1, 0.2, 0.3]) [OK]

Hint: Look for method named add_vector with id and vector [OK]

Common Mistakes:

• Using wrong method names like insert or push_vector
• Swapping argument order
• Missing parameter names

3. Given this code snippet, what will be the output?

query_vector = [0.5, 0.5]
results = vector_store.search(query_vector, top_k=2)
print(results)

Assuming the vector store contains vectors close to [0.5, 0.5] for ids 'docA' and 'docB'.

medium

A. [('docA', 0.98), ('docB', 0.95)]

B. [('docC', 0.50), ('docD', 0.45)]

C. []

D. Error: search method not found

Solution

1. Step 1: Understand search behavior

   The search method returns the top_k closest vectors by similarity score.

2. Step 2: Match expected results

   Since 'docA' and 'docB' are closest to [0.5, 0.5], they appear with high similarity scores like 0.98 and 0.95.

3. Final Answer:

   [('docA', 0.98), ('docB', 0.95)] -> Option A

4. Quick Check:

   Top matches with high similarity = [('docA', 0.98), ('docB', 0.95)] [OK]

Hint: Search returns closest vectors with highest similarity scores [OK]

Common Mistakes:

• Expecting unrelated documents in results
• Assuming empty list if vectors exist
• Thinking search method causes error

4. You run this code but get an error:

vector_store.add_vector([0.1, 0.2, 0.3], id='doc1')

What is the likely cause of the error?

medium

A. The add_vector method requires id first, then vector as keyword arguments

B. The vector argument should be named before id

C. Vectors cannot be lists, must be strings

D. The method add_vector does not exist

Solution

1. Step 1: Check method signature

   add_vector usually expects id first, then vector as keyword arguments.

2. Step 2: Identify argument order error

   Passing vector first without naming causes error; correct call is add_vector(id='doc1', vector=[0.1, 0.2, 0.3]).

3. Final Answer:

   The add_vector method requires id first, then vector as keyword arguments -> Option A

4. Quick Check:

   Correct argument order = The add_vector method requires id first, then vector as keyword arguments [OK]

Hint: Check argument order and names for add_vector [OK]

Common Mistakes:

• Passing vector before id without naming
• Thinking vectors must be strings
• Assuming method does not exist

5. You want your AI agent to remember past conversations using a vector store. Which approach best ensures it retrieves relevant past info quickly and accurately?

hard

A. Store conversation texts as raw strings and search by keyword matching

B. Convert conversation texts into vectors and search by similarity in the vector store

C. Save conversations in a plain text file and read line by line

D. Use a random vector for each conversation and pick one randomly

Solution

1. Step 1: Understand vector store advantage

   Vector stores allow searching by similarity, capturing meaning beyond exact words.

2. Step 2: Compare options for retrieval quality

   Convert conversation texts into vectors and search by similarity in the vector store converts texts to vectors and searches by similarity, enabling fast and accurate recall of related past conversations.

3. Final Answer:

   Convert conversation texts into vectors and search by similarity in the vector store -> Option B

4. Quick Check:

   Similarity search in vector store = best for relevant recall [OK]

Hint: Use vector similarity, not keyword or random picks [OK]

Common Mistakes:

• Relying on keyword matching only
• Using random vectors losing relevance
• Reading plain text files linearly