Memory retrieval strategies in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Memory retrieval strategies

Which metric matters for Memory retrieval strategies and WHY

For memory retrieval strategies in AI, the key metrics are Recall and Precision. Recall measures how many relevant memories the system successfully retrieves out of all possible relevant memories. Precision measures how many of the retrieved memories are actually relevant. High recall ensures the AI does not miss important information, while high precision ensures the AI does not retrieve irrelevant or noisy memories. Depending on the use case, one may prioritize recall (to avoid missing critical info) or precision (to avoid confusion from irrelevant data).

Confusion matrix for Memory retrieval

      | Retrieved Relevant | Retrieved Irrelevant |
      |-------------------|---------------------|
      | True Positives (TP) | False Positives (FP) |
      | False Negatives (FN)| True Negatives (TN)  |

      Example:
      TP = 80 (correct memories retrieved)
      FP = 20 (wrong memories retrieved)
      FN = 10 (relevant memories missed)
      TN = 90 (irrelevant memories correctly not retrieved)

      Total samples = TP + FP + FN + TN = 80 + 20 + 10 + 90 = 200
    

Precision vs Recall tradeoff with examples

Imagine an AI assistant recalling past conversations to answer a question.

• High Recall, Low Precision: The AI retrieves almost all relevant memories but also many irrelevant ones. This means it rarely misses important info but may confuse the answer with noise.
• High Precision, Low Recall: The AI retrieves only very confident memories, so most are relevant, but it may miss some important ones. This keeps answers clean but risks missing key details.

Choosing the right balance depends on the task. For critical decisions, high recall is better to avoid missing info. For quick answers, high precision avoids confusion.

What good vs bad metric values look like

Good memory retrieval strategy metrics:

• Recall above 0.85 means most relevant memories are found.
• Precision above 0.80 means most retrieved memories are relevant.
• F1 score (balance of precision and recall) above 0.80 is ideal.

Bad metrics examples:

• Recall below 0.50 means many relevant memories are missed.
• Precision below 0.50 means many irrelevant memories are retrieved.
• F1 score below 0.50 indicates poor overall retrieval quality.

Common pitfalls in memory retrieval metrics

• Accuracy paradox: High accuracy can be misleading if irrelevant memories dominate the dataset.
• Data leakage: If future memories leak into training, metrics will be unrealistically high.
• Overfitting: The system may memorize specific memories but fail to generalize to new queries.
• Ignoring recall: Focusing only on precision can cause missing important memories.
• Ignoring precision: Focusing only on recall can cause noisy, irrelevant retrievals.

Self-check question

Your memory retrieval model has 98% accuracy but only 12% recall on relevant memories. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most memories are irrelevant, so the model is good at ignoring irrelevant ones but misses almost all relevant memories (only 12% recall). This means it fails to retrieve important information, which is critical for memory retrieval tasks.

Key Result

Recall and precision are key metrics for memory retrieval; high recall avoids missing important memories, high precision avoids irrelevant noise.

Practice

1. What is the main purpose of memory retrieval strategies in agentic AI?

easy

A. To find stored information quickly and accurately

B. To create new data from scratch

C. To delete old information permanently

D. To slow down the AI's response time

Solution

1. Step 1: Understand the role of memory retrieval

   Memory retrieval strategies are designed to help AI find information it has stored before.

2. Step 2: Identify the main goal

   The goal is to do this quickly and accurately so the AI can respond well.

3. Final Answer:

   To find stored information quickly and accurately -> Option A

4. Quick Check:

   Memory retrieval = find info fast [OK]

Hint: Memory retrieval means finding stored info fast [OK]

Common Mistakes:

• Confusing retrieval with data creation
• Thinking retrieval deletes data
• Assuming retrieval slows AI down

2. Which of the following is the correct way to check if a memory item matches a query in Python?

easy

A. if memory_item === query:

B. if memory_item = query:

C. if memory_item == query:

D. if memory_item != query:

Solution

1. Step 1: Recall Python comparison syntax

   In Python, '==' checks if two values are equal.

2. Step 2: Identify correct equality check

   '=' is assignment, '===' is not valid in Python, '!=' means not equal.

3. Final Answer:

   if memory_item == query: -> Option C

4. Quick Check:

   Equality check in Python = '==' [OK]

Hint: Use '==' to compare values in Python [OK]

Common Mistakes:

• Using '=' instead of '==' for comparison
• Using '===' which is JavaScript syntax
• Confusing '!=' with equality check

3. Given the code below, what will be the output?

memory = ['apple', 'banana', 'cherry']
query = 'banana'
result = None
for item in memory:
    if item == query:
        result = item
        break
print(result)

medium

A. None

B. Error

C. 'apple'

D. 'banana'

Solution

1. Step 1: Loop through memory list

   The loop checks each item: 'apple', then 'banana', then 'cherry'.

2. Step 2: Check for match and break

   When 'banana' matches the query, result is set to 'banana' and loop stops.

3. Final Answer:

   'banana' -> Option D

4. Quick Check:

   Loop finds 'banana' and stops [OK]

Hint: Loop breaks on first match, returns that item [OK]

Common Mistakes:

• Assuming result stays None
• Thinking loop continues after match
• Confusing output with first list item

4. What is wrong with this memory retrieval code snippet?

memory = []
query = 'orange'
for item in memory:
    if item == query:
        print('Found')
    else:
        print('Not found')

medium

A. It prints 'Not found' multiple times incorrectly

B. It never prints anything if memory is empty

C. It causes a syntax error due to missing colon

D. It crashes because query is not defined

Solution

1. Step 1: Analyze empty memory list

   The for loop does not run at all if memory is empty.

2. Step 2: Check output behavior

   Since loop never runs, no print happens, so no indication of 'Not found'.

3. Final Answer:

   It never prints anything if memory is empty -> Option B

4. Quick Check:

   Empty list means no loop runs [OK]

Hint: Empty memory means loop skips, no output printed [OK]

Common Mistakes:

• Thinking 'Not found' prints once automatically
• Assuming syntax error without checking code
• Believing query is undefined

5. You want to improve a memory retrieval function to return 'Not found' if no match exists, even when memory is empty. Which code change achieves this best?

def retrieve(memory, query):
    for item in memory:
        if item == query:
            return item
    # What to add here?

hard

A. return 'Not found' after the loop

B. print('Not found') inside the loop

C. return None inside the loop

D. raise Exception('Not found') inside the loop

Solution

1. Step 1: Understand loop behavior

   If no item matches, loop finishes without returning.

2. Step 2: Add return after loop

   Returning 'Not found' after loop ensures function always returns a value.

3. Final Answer:

   return 'Not found' after the loop -> Option A

4. Quick Check:

   Return after loop handles no matches [OK]

Hint: Return 'Not found' after loop to handle no matches [OK]

Common Mistakes:

• Putting return inside loop causing premature exit
• Using print instead of return
• Raising exception unnecessarily