Dataset bias in vision in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Dataset bias in vision

Problem:You have trained an image classifier on a dataset where most images of cats are indoors and most images of dogs are outdoors. The model performs well on the training set but poorly on new images where cats and dogs appear in different environments.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is biased towards background cues (indoor/outdoor) instead of focusing on the animal features, causing poor generalization.

Your Task

Reduce dataset bias so the model focuses on the animal features, improving validation accuracy to at least 85% while keeping training accuracy below 92%.

You cannot collect new data.

You must use data augmentation or preprocessing techniques.

You must keep the same model architecture.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Define data augmentation to reduce background bias
train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.15,
    zoom_range=0.15,
    horizontal_flip=True,
    brightness_range=[0.7,1.3],
    validation_split=0.2
)

# Load training data with augmentation
train_generator = train_datagen.flow_from_directory(
    'dataset/train',
    target_size=(150, 150),
    batch_size=32,
    class_mode='binary',
    subset='training'
)

validation_generator = train_datagen.flow_from_directory(
    'dataset/train',
    target_size=(150, 150),
    batch_size=32,
    class_mode='binary',
    subset='validation'
)

# Define a simple CNN model
model = models.Sequential([
    layers.Conv2D(32, (3,3), activation='relu', input_shape=(150,150,3)),
    layers.MaxPooling2D(2,2),
    layers.Conv2D(64, (3,3), activation='relu'),
    layers.MaxPooling2D(2,2),
    layers.Conv2D(128, (3,3), activation='relu'),
    layers.MaxPooling2D(2,2),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train the model with augmented data
history = model.fit(
    train_generator,
    epochs=20,
    validation_data=validation_generator
)

# Output training and validation accuracy
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Added data augmentation with rotation, shifts, shear, zoom, flips, and brightness changes to reduce background bias.

Included dropout layer to reduce overfitting.

Kept the same CNN architecture but trained with augmented data.

Results Interpretation

Before: Training accuracy: 95%, Validation accuracy: 70% (overfitting and dataset bias)

After: Training accuracy: 90%, Validation accuracy: 87% (better generalization, less bias)

Using data augmentation helps the model learn features of the animals rather than relying on background cues, reducing dataset bias and improving validation accuracy.

Bonus Experiment

Try using feature visualization techniques to see what parts of the images the model focuses on before and after augmentation.

💡 Hint

Use Grad-CAM or saliency maps to visualize model attention on images.

Practice

1. What does dataset bias in computer vision mean?

easy

A. The data does not fairly represent all types of images or cases

B. The model always predicts perfectly on all images

C. The dataset is too large to process

D. The images are all black and white

Solution

1. Step 1: Understand dataset bias meaning

   Dataset bias means the data used to train a model does not cover all possible cases fairly.

2. Step 2: Compare options to definition

   Only The data does not fairly represent all types of images or cases describes this correctly. Other options describe unrelated issues.

3. Final Answer:

   The data does not fairly represent all types of images or cases -> Option A

4. Quick Check:

   Dataset bias = unfair data representation [OK]

Hint: Bias means data is not fair or balanced [OK]

Common Mistakes:

• Thinking bias means model is perfect
• Confusing bias with dataset size
• Assuming bias means image color

2. Which of the following is the correct way to check for dataset bias in a vision dataset using Python?

easy

A. Use random.shuffle(dataset) to fix bias

B. Use value_counts() on labels to see class distribution

C. Use len(dataset) without checking labels

D. Use print(dataset) only

Solution

1. Step 1: Identify method to check bias

   Checking class distribution with value_counts() helps find imbalance in labels.

2. Step 2: Evaluate other options

   Printing dataset or length alone doesn't show bias. Shuffling data doesn't check bias.

3. Final Answer:

   Use value_counts() on labels to see class distribution -> Option B

4. Quick Check:

   Check label counts = value_counts() [OK]

Hint: Check label counts to find bias [OK]

Common Mistakes:

• Only printing dataset without analysis
• Assuming dataset length shows bias
• Thinking shuffling fixes bias

3. Given this Python code snippet analyzing a vision dataset labels:

import pandas as pd
labels = ['cat', 'dog', 'cat', 'cat', 'dog', 'bird']
label_counts = pd.Series(labels).value_counts()
print(label_counts)

What is the output?

medium

A. bird 3
cat 2
dog 1

B. cat 2
dog 3
bird 1

C. cat 3
dog 2
bird 1

D. dog 3
cat 3
bird 3

Solution

1. Step 1: Count occurrences of each label

   Labels list has 'cat' 3 times, 'dog' 2 times, and 'bird' 1 time.

2. Step 2: Understand value_counts output

   value_counts() returns counts sorted descending by default.

3. Final Answer:

   cat 3\ndog 2\nbird 1 -> Option C

4. Quick Check:

   Count labels correctly = cat:3, dog:2, bird:1 [OK]

Hint: Count each label frequency carefully [OK]

Common Mistakes:

• Mixing counts of dog and cat
• Assuming alphabetical order instead of count order
• Miscounting occurrences

4. You have this code to check dataset bias:

labels = ['car', 'car', 'truck', 'car', 'truck']
counts = {}
for label in labels:
    counts[label] = counts.get(label, 0) + 1
print(counts)

But the output is {}. What is the likely error?

medium

A. The 'get' method is used incorrectly with wrong parameters

B. The print statement is outside the loop and missing indentation

C. The code is correct; output should be {'car': 3, 'truck': 2}

D. The dictionary 'counts' was reinitialized inside the loop

Solution

1. Step 1: Analyze code behavior

   If 'counts' is reset inside the loop, it will be empty after loop ends.

2. Step 2: Identify cause of empty output

   Reinitializing 'counts' inside loop clears previous counts, causing empty dict at print.

3. Final Answer:

   The dictionary 'counts' was reinitialized inside the loop -> Option D

4. Quick Check:

   Resetting dict inside loop empties counts [OK]

Hint: Check if dict is reset inside loop [OK]

Common Mistakes:

• Thinking print indentation causes empty output
• Assuming get() method is wrong
• Ignoring variable scope inside loop

5. You have a vision dataset with 90% images of cats and 10% dogs. Which method best reduces dataset bias to improve model fairness?

hard

A. Collect more dog images to balance classes

B. Remove all cat images to keep only dogs

C. Train model only on cat images

D. Ignore class imbalance and train as is

Solution

1. Step 1: Understand dataset imbalance problem

   Having 90% cats and 10% dogs causes bias favoring cats.

2. Step 2: Choose method to fix bias

   Collecting more dog images balances classes, reducing bias and improving fairness.

3. Final Answer:

   Collect more dog images to balance classes -> Option A

4. Quick Check:

   Balance classes by adding data [OK]

Hint: Balance classes by adding underrepresented data [OK]

Common Mistakes:

• Removing majority class loses useful data
• Training on one class ignores others
• Ignoring imbalance causes biased model