Why text generation solves real problems in Prompt Engineering / GenAI - Experiment to Prove It

Experiment - Why text generation solves real problems

Problem:We want to create a model that can generate helpful text responses for customer support questions automatically.

Current Metrics:Training loss: 0.05, Validation loss: 0.20, Training accuracy: 98%, Validation accuracy: 70%

Issue:The model is overfitting: it performs very well on training data but poorly on new, unseen questions.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only change model architecture and training settings.

Do not add more training data.

Keep the same dataset split.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with real data)
X_train, y_train = ...  # tokenized input and target sequences
X_val, y_val = ...

model = Sequential([
    Embedding(input_dim=10000, output_dim=64, input_length=100),
    LSTM(128, return_sequences=True),
    Dropout(0.3),
    LSTM(64),
    Dropout(0.3),
    Dense(10000, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=20, batch_size=64, validation_data=(X_val, y_val), callbacks=[early_stop])

Added dropout layers after LSTM layers to reduce overfitting.

Reduced learning rate from default to 0.001 for smoother training.

Added early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 70% (overfitting)

After: Training accuracy 90%, Validation accuracy 87% (better generalization)

Adding dropout and early stopping helps the model generalize better to new data by reducing overfitting, which improves real-world usefulness of text generation.

Bonus Experiment

Try using a smaller LSTM size or a simpler model to see if it also reduces overfitting.

💡 Hint

Simpler models often overfit less because they have fewer parameters to memorize training data.