Benchmark datasets in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Benchmark datasets

Problem:You want to train a simple image classifier using a popular benchmark dataset like CIFAR-10. The current model achieves 95% training accuracy but only 70% validation accuracy.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.85

Issue:The model is overfitting the training data and does not generalize well to new images.

Your Task

Reduce overfitting so that validation accuracy improves to at least 80% while keeping training accuracy below 90%.

Use the CIFAR-10 dataset only.

Keep the model architecture simple (e.g., a small CNN).

Do not increase the number of training epochs beyond 30.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Load CIFAR-10 dataset
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()

# Normalize pixel values
X_train, X_test = X_train / 255.0, X_test / 255.0

# Data augmentation
datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.1,
    height_shift_range=0.1,
    horizontal_flip=True
)
datagen.fit(X_train)

# Build a simple CNN model with dropout
model = models.Sequential([
    layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)),
    layers.MaxPooling2D((2, 2)),
    layers.Dropout(0.25),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Dropout(0.25),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model with data augmentation
history = model.fit(datagen.flow(X_train, y_train, batch_size=64),
                    epochs=30,
                    validation_data=(X_test, y_test),
                    verbose=2)

# Evaluate final model
final_train_loss, final_train_acc = model.evaluate(X_train, y_train, verbose=0)
final_val_loss, final_val_acc = model.evaluate(X_test, y_test, verbose=0)

print(f'Training accuracy: {final_train_acc*100:.2f}%')
print(f'Validation accuracy: {final_val_acc*100:.2f}%')

Added dropout layers after convolution and dense layers to reduce overfitting.

Applied data augmentation (rotation, shifts, flips) to increase training data variety.

Kept model architecture simple with two convolutional layers and one dense layer.

Used Adam optimizer with default learning rate and batch size of 64.

Limited training to 30 epochs.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.85

After: Training accuracy 88%, Validation accuracy 82%, Training loss 0.35, Validation loss 0.55

Adding dropout and data augmentation helps reduce overfitting, improving validation accuracy while slightly lowering training accuracy. This shows the importance of regularization and data variety in training models on benchmark datasets.

Bonus Experiment

Try using a different benchmark dataset like MNIST or Fashion-MNIST with the same model and techniques.

💡 Hint

Adjust input shape and number of classes accordingly, and observe if overfitting reduces similarly.

Practice

1. What is the main purpose of benchmark datasets in machine learning?

easy

A. To speed up model training by using smaller data

B. To provide a standard way to test and compare models

C. To store user data for training

D. To create new machine learning algorithms

Solution

1. Step 1: Understand the role of benchmark datasets

   Benchmark datasets are used to test machine learning models on the same data so results can be compared fairly.

2. Step 2: Identify the correct purpose

   They are not for creating algorithms or storing user data, but for evaluation and comparison.

3. Final Answer:

   To provide a standard way to test and compare models -> Option B

4. Quick Check:

   Benchmark datasets = standard test data [OK]

Hint: Benchmark datasets test models fairly with known data [OK]

Common Mistakes:

• Thinking benchmark datasets create algorithms
• Confusing benchmark datasets with training data
• Assuming benchmark datasets speed up training

2. Which of the following is the correct way to load the popular MNIST benchmark dataset in Python using TensorFlow?

easy

A. from tensorflow.keras.datasets import mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

B. import mnist
train_images, train_labels = mnist.load()

C. from sklearn.datasets import mnist
mnist.load()

D. load_mnist()

Solution

1. Step 1: Recall the TensorFlow MNIST loading syntax

   TensorFlow provides MNIST via keras.datasets with the load_data() method.

2. Step 2: Match the correct code snippet

   from tensorflow.keras.datasets import mnist (train_images, train_labels), (test_images, test_labels) = mnist.load_data() matches the correct import and loading syntax exactly.

3. Final Answer:

   from tensorflow.keras.datasets import mnist\n(train_images, train_labels), (test_images, test_labels) = mnist.load_data() -> Option A

4. Quick Check:

   TensorFlow MNIST load = keras.datasets.mnist.load_data() [OK]

Hint: TensorFlow MNIST loads with keras.datasets.mnist.load_data() [OK]

Common Mistakes:

• Using sklearn.datasets for MNIST (wrong library)
• Calling load() instead of load_data()
• Missing proper import statement

3. Given the following code snippet using the Iris dataset, what will be the output of print(data.target_names)?

from sklearn.datasets import load_iris
data = load_iris()
print(data.target_names)

medium

A. ['red', 'green', 'blue']

B. [0 1 2]

C. ['iris-setosa', 'iris-versicolor', 'iris-virginica']

D. ['setosa' 'versicolor' 'virginica']

Solution

1. Step 1: Understand the Iris dataset target names

   The Iris dataset target_names attribute contains the species names as numpy array strings without commas.

2. Step 2: Match the output format

   ['setosa' 'versicolor' 'virginica'] shows the correct array format with species names as strings without commas, matching sklearn output.

3. Final Answer:

   ['setosa' 'versicolor' 'virginica'] -> Option D

4. Quick Check:

   Iris target_names = species names array [OK]

Hint: Iris target_names shows species as array of strings [OK]

Common Mistakes:

• Confusing target_names with numeric labels
• Expecting commas inside numpy array print
• Using wrong species names

4. You try to load the CIFAR-10 dataset using this code but get an error:

from tensorflow.keras.datasets import cifar10
(train_images, train_labels), (test_images, test_labels) = cifar10.load()

What is the error and how to fix it?

medium

A. Error: SyntaxError due to missing parentheses, fix by adding () after load

B. Error: ImportError because cifar10 is not in keras.datasets, fix by installing extra package

C. Error: AttributeError because method is load_data(), fix by using cifar10.load_data()

D. No error, code runs fine

Solution

1. Step 1: Identify the method name for loading CIFAR-10

   The correct method to load CIFAR-10 in keras.datasets is load_data(), not load().

2. Step 2: Understand the error and fix

   Using cifar10.load() causes AttributeError. Changing to cifar10.load_data() fixes it.

3. Final Answer:

   Error: AttributeError because method is load_data(), fix by using cifar10.load_data() -> Option C

4. Quick Check:

   CIFAR-10 load method = load_data() [OK]

Hint: Use load_data() method to load datasets in keras.datasets [OK]

Common Mistakes:

• Using load() instead of load_data()
• Assuming cifar10 is not in keras.datasets
• Ignoring error message details

5. You want to compare two image classification models fairly. Which benchmark dataset should you choose and why?

hard

A. CIFAR-10 standard labeled image dataset for fair comparison

B. Unlabeled dataset for unsupervised learning

C. Small random dataset without standard labels

D. Single-class dataset to simplify training

Solution

1. Step 1: Understand the need for fair comparison

   Fair comparison requires a standard benchmark dataset with known labels and wide acceptance.

2. Step 2: Evaluate options for benchmark suitability

   CIFAR-10 is a popular benchmark with labeled images, suitable for comparing image classifiers fairly.

3. Final Answer:

   CIFAR-10 standard labeled image dataset for fair comparison -> Option A

4. Quick Check:

   Standard labeled dataset = fair model comparison [OK]

Hint: Choose standard labeled datasets for fair model comparison [OK]

Common Mistakes:

• Using unlabeled or small random datasets for comparison
• Choosing datasets with only one class
• Ignoring the need for standard benchmarks