Conversation management in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Conversation management

Problem:You have built a chatbot that answers questions but it often loses track of the conversation context after a few turns.

Current Metrics:Training accuracy: 95%, Validation accuracy: 60%, Loss: 0.4

Issue:The model overfits the training data and fails to maintain context, causing low validation accuracy and poor conversation flow.

Your Task

Reduce overfitting and improve the chatbot's ability to manage conversation context, aiming for validation accuracy above 80% while keeping training accuracy below 90%.

You cannot increase the model size significantly.

You must keep the training time reasonable (under 1 hour).

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout

# Sample data placeholders
X_train, y_train = ...  # Your training data
X_val, y_val = ...      # Your validation data

model = Sequential([
    Embedding(input_dim=10000, output_dim=64, input_length=20),
    LSTM(64, return_sequences=True),
    Dropout(0.3),
    LSTM(32),
    Dropout(0.3),
    Dense(64, activation='relu'),
    Dense(10, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_val, y_val))

Added dropout layers after LSTM layers to reduce overfitting.

Reduced LSTM units to keep model size manageable.

Set learning rate to 0.001 for stable training.

Kept training epochs to 20 to avoid overfitting.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 60%, Loss 0.4

After: Training accuracy 88%, Validation accuracy 82%, Loss 0.25

Adding dropout and tuning hyperparameters helps reduce overfitting and improves the model's ability to manage conversation context, leading to better validation accuracy.

Bonus Experiment

Try adding an attention mechanism to the model to further improve context understanding.

💡 Hint

Use TensorFlow's Attention layer or implement a custom attention mechanism to help the model focus on relevant parts of the conversation history.

Practice

1. What is the main purpose of conversation management in AI chat systems?

easy

A. To translate messages into different languages automatically

B. To speed up the AI's response time by skipping context

C. To delete old messages to save memory

D. To store chat messages and keep context for relevant replies

Solution

1. Step 1: Understand conversation management role

   Conversation management keeps track of messages to maintain context.

2. Step 2: Identify the benefit of context

   Context helps AI give replies that fit the ongoing chat naturally.

3. Final Answer:

   To store chat messages and keep context for relevant replies -> Option D

4. Quick Check:

   Conversation management = store messages + context [OK]

Hint: Remember: context means keeping chat history [OK]

Common Mistakes:

• Thinking it deletes messages instead of storing
• Confusing speed with context management
• Assuming it translates messages automatically

2. Which of the following is the correct way to represent a chat message in conversation management?

easy

A. {'text': 'Hello', 'role': 'user'}

B. ['Hello', 'user']

C. {'message': 'Hello', 'sender': 'bot'}

D. ('user', 'Hello')

Solution

1. Step 1: Identify standard message format

   Commonly, messages use keys like 'text' and 'role' to store content and sender.

2. Step 2: Compare options

   {'text': 'Hello', 'role': 'user'} uses {'text': ..., 'role': ...} which matches the typical format.

3. Final Answer:

   {'text': 'Hello', 'role': 'user'} -> Option A

4. Quick Check:

   Message = {'text', 'role'} format [OK]

Hint: Look for keys 'text' and 'role' in message dict [OK]

Common Mistakes:

• Using list or tuple instead of dict for messages
• Confusing 'sender' with 'role'
• Using wrong key names like 'message'

3. Given this conversation list:

messages = [
  {'role': 'user', 'text': 'Hi'},
  {'role': 'assistant', 'text': 'Hello! How can I help?'}
]

What will be the output of len(messages)?

medium

A. 1

B. 2

C. 0

D. Error

Solution

1. Step 1: Count the number of message dicts in the list

   There are two dictionaries inside the list representing two messages.

2. Step 2: Understand len() function on list

   len() returns the number of items in the list, which is 2 here.

3. Final Answer:

   2 -> Option B

4. Quick Check:

   len(messages) = 2 [OK]

Hint: Count items in list to find length [OK]

Common Mistakes:

• Counting keys inside dict instead of list items
• Assuming len() returns total characters
• Thinking len() causes error on list

4. What is wrong with this code snippet for adding a user message?

messages = []
messages.append({'role': 'user', 'message': 'Hello'})

medium

A. The list should be a dictionary instead

B. The role should be 'assistant' for user messages

C. The key 'message' should be 'text' to keep format consistent

D. append() cannot add dictionaries to a list

Solution

1. Step 1: Check message key naming

   The standard key for message content is 'text', not 'message'.

2. Step 2: Understand importance of consistent keys

   Using 'message' breaks the expected format and may cause errors later.

3. Final Answer:

   The key 'message' should be 'text' to keep format consistent -> Option C

4. Quick Check:

   Use 'text' key for message content [OK]

Hint: Use 'text' key for message content [OK]

Common Mistakes:

• Thinking append() can't add dicts
• Confusing roles for user and assistant
• Using wrong data structure for messages

5. You want to keep only the last 3 messages in a conversation to save memory. Which code correctly updates the messages list?

messages = [
  {'role': 'user', 'text': 'Hi'},
  {'role': 'assistant', 'text': 'Hello!'},
  {'role': 'user', 'text': 'How are you?'},
  {'role': 'assistant', 'text': 'Good, thanks!'}
]

hard

A. messages = messages[-3:]

B. messages = messages[:3]

C. messages = messages[3:]

D. messages = messages[:-3]

Solution

1. Step 1: Understand slicing to keep last 3 items

   Using negative index -3 in slicing keeps the last 3 messages.

2. Step 2: Check each option

   messages = messages[-3:] correctly slices from -3 to end, keeping last 3 messages.

3. Final Answer:

   messages = messages[-3:] -> Option A

4. Quick Check:

   Slice last 3 messages with [-3:] [OK]

Hint: Use negative slice [-3:] to keep last 3 items [OK]

Common Mistakes:

• Using [:3] keeps first 3, not last 3
• Using [3:] skips first 3, keeps last 1
• Using [:-3] removes last 3 instead of keeping