Why generative models create data in PyTorch - Experiment to Prove It

Experiment - Why generative models create data

Problem:You want to understand how a generative model can create new data similar to the training data. Currently, you have a simple generative model trained on handwritten digits, but it only memorizes the training images and does not create new variations.

Current Metrics:Training loss: 0.01, Validation loss: 0.5, Generated images look almost identical to training images.

Issue:The model is overfitting and memorizing training data instead of learning to generate new, diverse samples.

Your Task

Reduce overfitting so the generative model creates new, diverse images that look like handwritten digits but are not copies. Target: validation loss < 0.3 and generated images show variety.

Keep the model architecture similar (a simple Variational Autoencoder).

Do not increase training time beyond 50 epochs.

Hint 1

Hint 2

Hint 3

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
import torchvision

class VAE(nn.Module):
    def __init__(self, latent_dim=20):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 400)
        self.fc21 = nn.Linear(400, latent_dim)  # mean
        self.fc22 = nn.Linear(400, latent_dim)  # logvar
        self.fc3 = nn.Linear(latent_dim, 400)
        self.fc4 = nn.Linear(400, 28*28)
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()
        self.dropout = nn.Dropout(0.1)

    def encode(self, x):
        h1 = self.relu(self.fc1(x))
        h1 = self.dropout(h1)
        return self.fc21(h1), self.fc22(h1)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h3 = self.relu(self.fc3(z))
        return self.sigmoid(self.fc4(h3))

    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 28*28))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def loss_function(recon_x, x, mu, logvar):
    BCE = nn.functional.binary_cross_entropy(recon_x, x.view(-1, 28*28), reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD

transform = transforms.ToTensor()
dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
dataloader = DataLoader(dataset, batch_size=128, shuffle=True)

model = VAE(latent_dim=20)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

model.train()
for epoch in range(50):
    train_loss = 0
    for batch_idx, (data, _) in enumerate(dataloader):
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_function(recon_batch, data, mu, logvar)
        loss.backward()
        train_loss += loss.item()
        optimizer.step()
    avg_loss = train_loss / len(dataloader.dataset)
    if epoch % 10 == 0:
        print(f'Epoch {epoch}: Average loss: {avg_loss:.4f}')

# Generate new samples
model.eval()
with torch.no_grad():
    z = torch.randn(64, 20)
    sample = model.decode(z).cpu()
    sample = sample.view(64, 1, 28, 28)
    grid_img = torchvision.utils.make_grid(sample, nrow=8)
    plt.imshow(grid_img.permute(1, 2, 0))
    plt.title('Generated digits')
    plt.axis('off')
    plt.show()

Added dropout layer after first encoding layer to reduce overfitting.

Implemented Variational Autoencoder loss with KL divergence to encourage latent space regularization.

Set latent dimension to 20 to balance compression and detail.

Used Adam optimizer with learning rate 0.001 and trained for 50 epochs.

Results Interpretation

Before: Training loss 0.01, Validation loss 0.5, Generated images are copies of training data.

After: Training loss 0.12, Validation loss 0.28, Generated images are new and diverse handwritten digits.

Using a Variational Autoencoder with KL divergence loss and dropout helps the model learn a smooth latent space. This reduces overfitting and allows the model to create new, varied data instead of memorizing.

Bonus Experiment

Try increasing the latent dimension to 50 and observe how it affects the diversity and quality of generated images.

💡 Hint

A larger latent space can capture more details but may require more training or stronger regularization to avoid overfitting.

Practice

1. Why do generative models create new data in machine learning?

easy

A. To speed up training by skipping data

B. To delete existing data from the dataset

C. To only classify data into categories

D. To mimic real examples and generate similar data

Solution

1. Step 1: Understand the purpose of generative models

   Generative models learn patterns from real data to create new, similar data.

2. Step 2: Identify the correct purpose from options

   Only To mimic real examples and generate similar data describes creating new data that mimics real examples, which is the main goal.

3. Final Answer:

   To mimic real examples and generate similar data -> Option D

4. Quick Check:

   Generative models create data = mimic real examples [OK]

Hint: Generative models create data similar to real examples [OK]

Common Mistakes:

• Confusing generative models with classifiers
• Thinking generative models delete data
• Assuming generative models speed up training by skipping data

2. Which PyTorch code snippet correctly initializes a simple generative model layer?

easy

A. torch.nn.Linear(100, 256)

B. torch.nn.Conv2d(3, 64, kernel_size=3)

C. torch.optim.SGD(model.parameters(), lr=0.01)

D. torch.nn.CrossEntropyLoss()

Solution

1. Step 1: Identify generative model layer types

   Generative models often use linear layers to transform random input vectors into data representations.

2. Step 2: Match code to layer initialization

   torch.nn.Linear(100, 256) correctly initializes a linear layer with input size 100 and output size 256, suitable for generative models.

3. Final Answer:

   torch.nn.Linear(100, 256) -> Option A

4. Quick Check:

   Linear layer init = torch.nn.Linear(100, 256) [OK]

Hint: Linear layers often start generative models [OK]

Common Mistakes:

• Choosing optimizer or loss function as layer
• Confusing convolutional layers with simple linear layers
• Using loss functions as model layers

3. Given this PyTorch code, what is the shape of the generated data tensor?

import torch
noise = torch.randn(16, 100)
generator = torch.nn.Linear(100, 784)
output = generator(noise)

medium

A. (16, 784)

B. (16, 100)

C. (100, 784)

D. (784, 16)

Solution

1. Step 1: Understand input and output shapes

   The noise tensor has shape (16, 100), meaning 16 samples each with 100 features. The linear layer maps 100 features to 784 features.

2. Step 2: Calculate output shape after linear layer

   Applying the linear layer to noise results in shape (16, 784), preserving batch size and changing feature size.

3. Final Answer:

   (16, 784) -> Option A

4. Quick Check:

   Output shape = (batch_size, output_features) = (16, 784) [OK]

Hint: Output shape matches batch size and output features [OK]

Common Mistakes:

• Mixing batch size and feature dimensions
• Confusing input and output shapes
• Transposing shapes incorrectly

4. Find the error in this PyTorch code snippet for generating data:

import torch
noise = torch.randn(32, 50)
generator = torch.nn.Linear(100, 784)
output = generator(noise)

medium

A. Output tensor shape is correct

B. Generator layer should be torch.nn.Conv2d

C. Noise tensor has wrong shape for the generator input

D. Missing optimizer initialization

Solution

1. Step 1: Check input shape compatibility

   The generator expects input with 100 features, but noise has 50 features per sample, causing a shape mismatch error.

2. Step 2: Identify the error cause

   Because noise shape (32, 50) does not match expected input (batch_size, 100), the code will fail at runtime.

3. Final Answer:

   Noise tensor has wrong shape for the generator input -> Option C

4. Quick Check:

   Input shape mismatch = noise shape wrong [OK]

Hint: Input features must match layer input size [OK]

Common Mistakes:

• Ignoring input-output shape mismatch
• Thinking convolutional layers are always needed
• Confusing optimizer with model layers

5. Why do generative models use random noise as input when creating new data?

hard

A. To speed up training by skipping data

B. To introduce variety and generate diverse outputs

C. To reduce the size of the model

D. To make the model deterministic and fixed

Solution

1. Step 1: Understand the role of random noise in generative models

   Random noise acts as a seed to produce different outputs, allowing the model to generate varied data samples.

2. Step 2: Match the purpose with the correct option

   To introduce variety and generate diverse outputs correctly states that noise introduces variety and diversity in generated data.

3. Final Answer:

   To introduce variety and generate diverse outputs -> Option B

4. Quick Check:

   Noise input = variety in outputs [OK]

Hint: Noise input creates diverse generated data [OK]

Common Mistakes:

• Thinking noise speeds up training
• Assuming noise reduces model size
• Believing noise makes output fixed