Short-term memory (conversation context) in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Short-term memory (conversation context)

Which metric matters for Short-term memory (conversation context) and WHY

For short-term memory in conversation AI, context retention accuracy is key. This measures how well the model remembers recent conversation details to respond correctly. Metrics like precision and recall on context-dependent responses help check if the model uses memory properly. Good context use means fewer mistakes and more relevant replies.

Confusion matrix for context understanding

    | Predicted Correct Context | Predicted Incorrect Context |
    |---------------------------|-----------------------------|
    | True Positive (TP) = 80    | False Positive (FP) = 10     |
    | False Negative (FN) = 15   | True Negative (TN) = 95      |

    Total samples = 80 + 10 + 15 + 95 = 200

    Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
    Recall = TP / (TP + FN) = 80 / (80 + 15) = 0.84
    F1 Score = 2 * (0.89 * 0.84) / (0.89 + 0.84) ≈ 0.86
    

This matrix shows how often the model correctly uses short-term memory (TP), mistakes context (FP), misses context (FN), or correctly ignores irrelevant context (TN).

Precision vs Recall tradeoff in conversation context

If the model has high precision, it means when it uses memory, it is usually correct. This avoids confusing or wrong replies. But if recall is low, the model forgets some important context, missing chances to respond well.

For example, in a customer chat, high recall ensures the model remembers all recent questions, avoiding repeated answers. High precision avoids mixing up different topics. Balancing both is important for smooth conversations.

What good vs bad metric values look like

Good values: Precision and recall above 0.85 show the model remembers and uses context well. F1 score near 0.9 means balanced performance.

Bad values: Precision or recall below 0.6 means the model often forgets or misuses context. This leads to confusing or irrelevant replies, hurting user experience.

Common pitfalls in evaluating short-term memory

• Accuracy paradox: High overall accuracy can hide poor context use if most replies don't need memory.
• Data leakage: If test data repeats conversation parts from training, metrics look better than real.
• Overfitting: Model may memorize fixed conversation patterns but fail on new topics.
• Ignoring recall: Missing context details can be worse than occasional wrong context use.

Self-check question

Your conversation AI model has 98% accuracy but only 12% recall on context-dependent replies. Is it good for production?

Answer: No. The low recall means the model forgets most important recent context. Even with high accuracy, it will often miss key details, causing poor user experience. Improving recall is critical before production.

Key Result

Precision and recall above 0.85 indicate good short-term memory use in conversation AI, balancing correct context use and coverage.

Practice

1. What is the main purpose of short-term memory in an AI conversation?

easy

A. To remember recent messages and keep the conversation connected

B. To store all past conversations permanently

C. To delete irrelevant messages immediately

D. To speed up the AI's processing by ignoring context

Solution

1. Step 1: Understand short-term memory role

   Short-term memory stores recent conversation parts to keep context.

2. Step 2: Compare options with this role

   Only To remember recent messages and keep the conversation connected matches this purpose; others describe different or incorrect functions.

3. Final Answer:

   To remember recent messages and keep the conversation connected -> Option A

4. Quick Check:

   Short-term memory = recent context [OK]

Hint: Short-term memory = recent messages stored [OK]

Common Mistakes:

• Confusing short-term with long-term memory
• Thinking it stores all past conversations
• Believing it deletes messages immediately

2. Which of the following is the correct way to represent short-term memory storing the last 3 messages in Python?

easy

A. short_term_memory = messages[0]

B. short_term_memory = messages[:3]

C. short_term_memory = messages[3:]

D. short_term_memory = messages[-3:]

Solution

1. Step 1: Understand Python list slicing for last 3 items

   Using messages[-3:] gets the last 3 messages from the list.

2. Step 2: Check other options

   messages[:3] gets first 3, messages[3:] gets from 4th to end, messages[0] gets only first message.

3. Final Answer:

   short_term_memory = messages[-3:] -> Option D

4. Quick Check:

   Last 3 messages slice = messages[-3:] [OK]

Hint: Negative slice gets last items in list [OK]

Common Mistakes:

• Using positive slice for last items
• Selecting only one message instead of three
• Confusing start and end indices

3. Given the code below, what will be the output of print(short_term_memory)?

messages = ['Hi', 'How are you?', 'I am fine', 'What about you?', 'Good!']
short_term_memory = messages[-2:]
print(short_term_memory)

medium

A. ['Hi', 'How are you?']

B. ['I am fine', 'What about you?']

C. ['What about you?', 'Good!']

D. ['Good!']

Solution

1. Step 1: Understand list slicing with negative indices

   messages[-2:] selects the last two items from the list.

2. Step 2: Identify last two messages

   The last two messages are 'What about you?' and 'Good!'.

3. Final Answer:

   ['What about you?', 'Good!'] -> Option C

4. Quick Check:

   messages[-2:] = last two messages [OK]

Hint: Negative slice picks last elements [OK]

Common Mistakes:

• Selecting wrong slice range
• Confusing order of messages
• Printing only one message instead of two

4. The following code is intended to keep only the last 3 messages in short-term memory, but it has a bug. What is the bug?

messages = ['Hello', 'What is AI?', 'Tell me more', 'Thanks']
short_term_memory = messages[3:]
print(short_term_memory)

medium

A. It causes an IndexError

B. It keeps only the last message instead of last three

C. It keeps the first three messages instead of last three

D. It clears the list completely

Solution

1. Step 1: Analyze the slice messages[3:]

   This slice starts at index 3 and goes to the end, so it keeps only the last message 'Thanks'.

2. Step 2: Compare with intended behavior

   The goal was to keep last 3 messages, but this code keeps only one message.

3. Final Answer:

   It keeps only the last message instead of last three -> Option B

4. Quick Check:

   messages[3:] = last message only [OK]

Hint: Check slice start index carefully [OK]

Common Mistakes:

• Assuming slice keeps last 3 messages
• Expecting an error when none occurs
• Confusing slice start and end

5. You want an AI agent to remember the last 4 messages in a conversation to keep context. The conversation messages are stored in a list called chat_history. Which code snippet correctly updates the short-term memory to always hold the last 4 messages after adding a new message new_msg?

hard

A. chat_history.append(new_msg)
short_term_memory = chat_history[-4:]

B. short_term_memory = chat_history[:4]
chat_history.append(new_msg)

C. short_term_memory = chat_history[-4:]
chat_history.append(new_msg)

D. chat_history = chat_history[-4:]
short_term_memory = new_msg

Solution

1. Step 1: Add new message to chat_history first

   Appending new_msg to chat_history updates the conversation.

2. Step 2: Slice last 4 messages for short-term memory

   Using chat_history[-4:] gets the last 4 messages including the new one.

3. Final Answer:

   chat_history.append(new_msg) short_term_memory = chat_history[-4:] -> Option A

4. Quick Check:

   Append then slice last 4 messages [OK]

Hint: Append first, then slice last 4 [OK]

Common Mistakes:

• Slicing before appending new message
• Assigning new message alone as memory
• Slicing first 4 messages instead of last 4