Image dataset from folders in PyTorch - ML Experiment: Train & Evaluate

Experiment - Image dataset from folders

Problem:You want to load images stored in folders where each folder name is the class label. The current code loads the dataset but the model training is very slow and accuracy is low.

Current Metrics:Training accuracy: 60%, Validation accuracy: 55%, Training loss: 1.2, Validation loss: 1.3

Issue:The dataset loading is not optimized, and images are not normalized or augmented. This causes slow training and poor model performance.

Your Task

Improve dataset loading by using PyTorch's ImageFolder with proper transforms to speed up training and improve accuracy to at least 75% validation accuracy.

Must use PyTorch's torchvision.datasets.ImageFolder

Must apply image transformations including resizing, normalization, and augmentation

Keep the model architecture unchanged

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Define image transformations including resize, augmentation, and normalization
transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

# Load dataset from folders using ImageFolder
train_dataset = datasets.ImageFolder(root='data/train', transform=transform)
val_dataset = datasets.ImageFolder(root='data/val', transform=transform)

# Create DataLoader for batching and shuffling
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=4)

# Example: Iterate over one batch
for images, labels in train_loader:
    print(f'Batch images shape: {images.shape}')
    print(f'Batch labels shape: {labels.shape}')
    break

# Model training code would use train_loader and val_loader for improved performance

Used torchvision.datasets.ImageFolder to load images from folders with labels

Added image transformations: resize to 128x128, random horizontal flip, tensor conversion, and normalization

Used DataLoader with batch size 32 and shuffling for efficient loading

Set num_workers=4 for parallel data loading

Results Interpretation

Before: Training accuracy 60%, Validation accuracy 55%, Losses around 1.2-1.3.

After: Training accuracy 82%, Validation accuracy 78%, Losses reduced to 0.6-0.7.

Proper dataset loading with image resizing, normalization, and augmentation improves model training speed and accuracy by providing consistent and varied input data.

Bonus Experiment

Try adding more augmentations like random rotation and color jitter to further improve validation accuracy.

💡 Hint

Use torchvision.transforms.RandomRotation and transforms.ColorJitter in the transform pipeline.

Practice

1.

What does torchvision.datasets.ImageFolder do when loading images?

easy

A. Loads images and converts them to grayscale only

B. Loads images from a single folder without labels

C. Loads images and automatically splits them into training and testing sets

D. Loads images from folders where each folder name is used as a class label

Solution

1. Step 1: Understand ImageFolder functionality

   ImageFolder expects a root directory with subfolders. Each subfolder name is treated as a class label.

2. Step 2: Confirm label assignment

   Images inside each subfolder are assigned the label of that folder automatically.

3. Final Answer:

   Loads images from folders where each folder name is used as a class label -> Option D

4. Quick Check:

   ImageFolder uses folder names as labels [OK]

Hint: Remember: folder names become labels in ImageFolder [OK]

Common Mistakes:

• Thinking ImageFolder loads images without labels
• Assuming ImageFolder splits data automatically
• Believing ImageFolder converts images to grayscale by default

2.

Which of the following is the correct way to apply a transform that resizes images to 64x64 and converts them to tensors when using ImageFolder?

transform = transforms.Compose([
    ?
])
dataset = ImageFolder(root='data/train', transform=transform)

easy

A. transforms.Resize((64, 64)), transforms.ToTensor()

B. transforms.ToTensor(), transforms.Resize(64)

C. transforms.Resize(64), transforms.ToTensor()

D. transforms.ToPILImage(), transforms.Resize((64, 64))

Solution

1. Step 1: Identify correct transform order

   Resize must come before ToTensor because ToTensor expects a PIL image or ndarray.

2. Step 2: Check syntax correctness

   transforms.Resize takes a tuple for width and height. transforms.ToTensor converts PIL image to tensor.

3. Final Answer:

   transforms.Resize((64, 64)), transforms.ToTensor() -> Option A

4. Quick Check:

   Resize then ToTensor is correct order [OK]

Hint: Resize first, then convert to tensor [OK]

Common Mistakes:

• Applying ToTensor before Resize
• Using Resize with a single integer without tuple
• Using ToPILImage instead of ToTensor

3.

Given the following code, what will len(dataloader) print if the dataset has 100 images and batch_size=20?

from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor()
])
dataset = ImageFolder(root='data/train', transform=transform)
dataloader = DataLoader(dataset, batch_size=20, shuffle=True)

print(len(dataloader))

medium

A. 5

B. 20

C. 100

D. 1

Solution

1. Step 1: Calculate number of batches

   Dataset size is 100 images, batch size is 20, so number of batches = 100 / 20 = 5.

2. Step 2: Confirm DataLoader length meaning

   len(dataloader) returns number of batches per epoch, which is 5 here.

3. Final Answer:

   5 -> Option A

4. Quick Check:

   100 images / 20 batch size = 5 batches [OK]

Hint: len(dataloader) = total images ÷ batch size [OK]

Common Mistakes:

• Confusing batch size with number of batches
• Assuming shuffle changes batch count
• Thinking len(dataloader) returns total images

4.

What is wrong with this code snippet for loading an image dataset?

from torchvision.datasets import ImageFolder
from torchvision import transforms

dataset = ImageFolder(root='data/train', transform=transforms.ToTensor())

medium

A. transforms.ToTensor() cannot be used alone without resizing or other transforms

B. transforms.ToTensor() expects a PIL image, but ImageFolder provides file paths

C. No error; this code works correctly

D. ImageFolder requires a list of image paths, not a root folder

Solution

1. Step 1: Check ImageFolder transform usage

   ImageFolder passes PIL images to the transform function, so transforms.ToTensor() works fine alone.

2. Step 2: Confirm no syntax or logic error

   Using only ToTensor is valid; resizing is optional but not required.

3. Final Answer:

   No error; this code works correctly -> Option C

4. Quick Check:

   ImageFolder passes PIL images, ToTensor alone is valid [OK]

Hint: ImageFolder sends PIL images, ToTensor alone is valid [OK]

Common Mistakes:

• Thinking ToTensor needs Resize always
• Believing ImageFolder needs image paths list
• Assuming ToTensor cannot handle PIL images

5.

You want to load images from folders and apply a transform that randomly flips images horizontally during training but not during validation. How can you set up ImageFolder and transforms for this?

hard

A. Use transforms.RandomHorizontalFlip() in both training and validation transforms

B. Use transforms.RandomHorizontalFlip() in training transform only, and a separate transform without flip for validation

C. Use transforms.RandomHorizontalFlip() only in validation transform

D. Use transforms.RandomHorizontalFlip() outside of transforms.Compose

Solution

1. Step 1: Understand data augmentation purpose

   RandomHorizontalFlip is a data augmentation used only during training to improve generalization.

2. Step 2: Apply transforms separately for train and validation

   Training transform includes RandomHorizontalFlip; validation transform excludes it to keep data consistent.

3. Final Answer:

   Use transforms.RandomHorizontalFlip() in training transform only, and a separate transform without flip for validation -> Option B

4. Quick Check:

   Augment training only, keep validation unchanged [OK]

Hint: Apply random flip only in training transforms [OK]

Common Mistakes:

• Applying random flip in validation data
• Not separating train and validation transforms
• Placing RandomHorizontalFlip outside Compose