Why custom data pipelines handle real data in PyTorch

Introduction

Custom data pipelines help prepare and organize real data so models can learn from it easily and correctly.

When your data is messy or in different formats like images, text, or numbers.

When you need to load data in small parts because it is too big to fit in memory.

When you want to change or improve data on the fly, like resizing images or adding noise.

When you want to shuffle or batch data to help the model learn better.

When you want to combine data from multiple sources into one stream for training.

Syntax

PyTorch

import torch

class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, data_paths, transform=None):
        self.data_paths = data_paths
        self.transform = transform

    def __len__(self):
        return len(self.data_paths)

    def __getitem__(self, idx):
        data = load_data(self.data_paths[idx])
        if self.transform:
            data = self.transform(data)
        return data

from torch.utils.data import DataLoader

dataset = CustomDataset(data_paths, transform=some_transform)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

The __getitem__ method loads and processes one data item at a time.

Transforms help change data during loading, like resizing or normalizing images.

Examples

This example shows a dataset for text data that converts each text to lowercase.

PyTorch

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, texts):
        self.texts = texts

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        return self.texts[idx].lower()

This example applies image resizing and converts images to tensors during loading.

PyTorch

from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.ToTensor()
])

dataset = CustomDataset(image_paths, transform=transform)

Sample Model

This program creates a custom dataset for images, applies resizing and tensor conversion, then loads data in batches.

PyTorch

import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
from PIL import Image

class CustomDataset(Dataset):
    def __init__(self, image_paths, transform=None):
        self.image_paths = image_paths
        self.transform = transform

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        image = Image.open(self.image_paths[idx])
        if self.transform:
            image = self.transform(image)
        return image

# Sample image paths (replace with actual image files in practice)
image_paths = ["sample1.jpg", "sample2.jpg", "sample3.jpg"]

# Define transforms
transform = transforms.Compose([
    transforms.Resize((64, 64)),
    transforms.ToTensor()
])

# Create dataset and dataloader
dataset = CustomDataset(image_paths, transform=transform)
dataloader = DataLoader(dataset, batch_size=2, shuffle=True)

# Iterate over data
for batch in dataloader:
    print(batch.shape)

Important Notes

Custom pipelines let you control how data is loaded and changed before training.

Using DataLoader helps handle batching and shuffling automatically.

Transforms make it easy to prepare data consistently.

Summary

Custom data pipelines organize and prepare real data for models.

They handle loading, transforming, batching, and shuffling data.

This helps models learn better and makes training easier.