Chatbot development basics in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Chatbot development basics

Problem:You want to build a simple chatbot that can answer basic questions about a small set of topics. The current chatbot model answers training questions very well but performs poorly on new questions, showing signs of overfitting.

Current Metrics:Training accuracy: 98%, Validation accuracy: 65%, Training loss: 0.05, Validation loss: 0.9

Issue:The chatbot model is overfitting the training data, causing low accuracy on new, unseen questions.

Your Task

Reduce overfitting so that validation accuracy improves to at least 80%, while keeping training accuracy below 95% to ensure the model generalizes better.

You can only change model architecture and training hyperparameters.

You cannot add more training data.

You must keep the chatbot's response time reasonable (no very large models).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with actual data)
X_train, y_train = ...  # training features and labels
X_val, y_val = ...      # validation features and labels

# Define a simpler model with dropout
model = Sequential([
    Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
    Dropout(0.3),
    Dense(32, activation='relu'),
    Dropout(0.3),
    Dense(y_train.shape[1], activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Train the model
history = model.fit(X_train, y_train,
                    epochs=50,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added dropout layers with 30% rate to reduce overfitting.

Reduced model size from larger layers to 64 and 32 units.

Added early stopping to stop training when validation loss stops improving.

Set learning rate to 0.001 for stable training.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 65%, Training loss 0.05, Validation loss 0.9

After: Training accuracy 92%, Validation accuracy 82%, Training loss 0.18, Validation loss 0.35

Adding dropout and early stopping helped the chatbot model avoid memorizing training data and improved its ability to answer new questions correctly.

Bonus Experiment

Try using data augmentation techniques on the chatbot's training questions to further improve validation accuracy.

💡 Hint

Create paraphrased versions of training questions to increase data variety without collecting new data.

Practice

1. What is the main purpose of a chatbot in simple terms?

easy

A. To help computers talk with people easily

B. To store large amounts of data

C. To create images from text

D. To run complex math calculations

Solution

1. Step 1: Understand chatbot function

   A chatbot is designed to communicate with people using text or voice.

2. Step 2: Match purpose with options

   Only To help computers talk with people easily describes helping computers talk with people easily.

3. Final Answer:

   To help computers talk with people easily -> Option A

4. Quick Check:

   Chatbot purpose = talk with people [OK]

Hint: Chatbots are for chatting, not storing or calculating [OK]

Common Mistakes:

• Confusing chatbots with data storage systems
• Thinking chatbots create images
• Assuming chatbots do math calculations

2. Which of the following is the correct way to define a simple chatbot response in Python?

easy

A. response = (hello: 'Hi there!')

B. response = {'hello': 'Hi there!'}

C. response = ['hello' => 'Hi there!']

D. response = 'hello' = 'Hi there!'

Solution

1. Step 1: Recall Python dictionary syntax

   Python uses curly braces {} with key: value pairs for dictionaries.

2. Step 2: Check each option

   response = {'hello': 'Hi there!'} uses correct syntax with {'hello': 'Hi there!'}; others use invalid syntax.

3. Final Answer:

   response = {'hello': 'Hi there!'} -> Option B

4. Quick Check:

   Python dict = {'key': 'value'} [OK]

Hint: Python dict uses curly braces and colon for key-value [OK]

Common Mistakes:

• Using => instead of : in Python dictionaries
• Using parentheses instead of braces
• Trying to assign string with = inside quotes

3. What will be the output of this Python code snippet for a chatbot?

responses = {'hi': 'Hello!', 'bye': 'Goodbye!'}
user_input = 'hi'
print(responses.get(user_input, 'I do not understand'))

medium

A. Error

B. Goodbye!

C. I do not understand

D. Hello!

Solution

1. Step 1: Understand dictionary get method

   responses.get(user_input, default) returns value for key or default if key missing.

2. Step 2: Check user_input key in dictionary

   user_input is 'hi', which exists in responses with value 'Hello!'.

3. Final Answer:

   Hello! -> Option D

4. Quick Check:

   Key 'hi' found = 'Hello!' [OK]

Hint: dict.get(key, default) returns value or default if missing [OK]

Common Mistakes:

• Assuming default message prints even if key exists
• Confusing keys 'hi' and 'bye'
• Expecting an error from get method

4. Identify the error in this chatbot code snippet:

responses = {'hello': 'Hi!'}
user_input = input('Say something: ')
print(responses[user_input])

medium

A. print statement is incorrect

B. Syntax error in dictionary definition

C. Missing default response if input not in dictionary

D. input() function is not allowed in chatbot

Solution

1. Step 1: Analyze dictionary access

   Accessing responses[user_input] causes error if user_input key not found.

2. Step 2: Check for default handling

   Code lacks default fallback; should use get() or try-except to avoid crash.

3. Final Answer:

   Missing default response if input not in dictionary -> Option C

4. Quick Check:

   Direct dict access needs key check [OK]

Hint: Use dict.get() to avoid key errors from unknown input [OK]

Common Mistakes:

• Thinking input() is disallowed in chatbot
• Believing dictionary syntax is wrong
• Assuming print statement is incorrect

5. You want your chatbot to answer "Good morning!" when the user says "morning" or "good morning". Which Python code snippet correctly handles this?

hard

A. responses = {'morning': 'Good morning!', 'good morning': 'Good morning!'}

B. responses = {'morning' or 'good morning': 'Good morning!'}

C. responses = {'morning' & 'good morning': 'Good morning!'}

D. responses = {'morning' + 'good morning': 'Good morning!'}

Solution

1. Step 1: Understand dictionary keys for multiple inputs

   Each key must be separate to match different user inputs.

2. Step 2: Evaluate options for correct syntax

   responses = {'morning': 'Good morning!', 'good morning': 'Good morning!'} defines two keys separately; others use invalid Python expressions as keys.

3. Final Answer:

   responses = {'morning': 'Good morning!', 'good morning': 'Good morning!'} -> Option A

4. Quick Check:

   Separate keys for inputs = responses = {'morning': 'Good morning!', 'good morning': 'Good morning!'} [OK]

Hint: Use separate keys for each input phrase in dictionary [OK]

Common Mistakes:

• Trying to combine keys with or/&/+ operators
• Using invalid syntax for dictionary keys
• Assuming one key can match multiple phrases