Chatbot development basics in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Chatbot development basics

Problem:You want to build a simple chatbot that can answer basic questions about a small set of topics. The current chatbot model answers training questions very well but performs poorly on new questions, showing signs of overfitting.

Current Metrics:Training accuracy: 98%, Validation accuracy: 65%, Training loss: 0.05, Validation loss: 0.9

Issue:The chatbot model is overfitting the training data, causing low accuracy on new, unseen questions.

Your Task

Reduce overfitting so that validation accuracy improves to at least 80%, while keeping training accuracy below 95% to ensure the model generalizes better.

You can only change model architecture and training hyperparameters.

You cannot add more training data.

You must keep the chatbot's response time reasonable (no very large models).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with actual data)
X_train, y_train = ...  # training features and labels
X_val, y_val = ...      # validation features and labels

# Define a simpler model with dropout
model = Sequential([
    Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
    Dropout(0.3),
    Dense(32, activation='relu'),
    Dropout(0.3),
    Dense(y_train.shape[1], activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Train the model
history = model.fit(X_train, y_train,
                    epochs=50,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added dropout layers with 30% rate to reduce overfitting.

Reduced model size from larger layers to 64 and 32 units.

Added early stopping to stop training when validation loss stops improving.

Set learning rate to 0.001 for stable training.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 65%, Training loss 0.05, Validation loss 0.9

After: Training accuracy 92%, Validation accuracy 82%, Training loss 0.18, Validation loss 0.35

Adding dropout and early stopping helped the chatbot model avoid memorizing training data and improved its ability to answer new questions correctly.

Bonus Experiment

Try using data augmentation techniques on the chatbot's training questions to further improve validation accuracy.

💡 Hint

Create paraphrased versions of training questions to increase data variety without collecting new data.