Variational Autoencoder in PyTorch

Introduction

A Variational Autoencoder (VAE) helps us learn how to create new data similar to what we have by compressing and then rebuilding it in a smart way.

When you want to generate new images that look like your training pictures.

When you want to reduce the size of data but keep important features.

When you want to learn hidden patterns in data for tasks like anomaly detection.

When you want to create smooth transitions between different data points, like morphing faces.

When you want to explore creative AI applications like generating music or text.

Syntax

PyTorch

class VAE(nn.Module):
    def __init__(self):
        super().__init__()
        # Define encoder layers
        # Define layers to get mean and log variance
        # Define decoder layers

    def encode(self, x):
        # Pass x through encoder
        # Return mean and log variance

    def reparameterize(self, mu, logvar):
        # Sample from normal distribution using mu and logvar

    def decode(self, z):
        # Pass z through decoder

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

The encoder compresses input into two values: mean (mu) and log variance (logvar).

The reparameterization trick allows backpropagation through random sampling.

Examples

This code samples a point from the distribution defined by mu and logvar.

PyTorch

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5 * logvar)
    eps = torch.randn_like(std)
    return mu + eps * std

This loss combines reconstruction error and a penalty to keep the distribution close to normal.

PyTorch

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x, reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD

Sample Model

This program trains a simple VAE on MNIST digits for one epoch and prints the loss.

PyTorch

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

class VAE(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 400)
        self.fc21 = nn.Linear(400, 20)  # mu
        self.fc22 = nn.Linear(400, 20)  # logvar
        self.fc3 = nn.Linear(20, 400)
        self.fc4 = nn.Linear(400, 28*28)

    def encode(self, x):
        h1 = F.relu(self.fc1(x))
        return self.fc21(h1), self.fc22(h1)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h3 = F.relu(self.fc3(z))
        return torch.sigmoid(self.fc4(h3))

    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 28*28))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 28*28), reduction='sum')
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD

# Load MNIST dataset
train_loader = DataLoader(
    datasets.MNIST('.', train=True, download=True,
                   transform=transforms.ToTensor()),
    batch_size=128, shuffle=True)

model = VAE()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

model.train()
for epoch in range(1):  # 1 epoch for demo
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader):
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_function(recon_batch, data, mu, logvar)
        loss.backward()
        train_loss += loss.item()
        optimizer.step()
        if batch_idx == 0:
            print(f'Train Epoch: {epoch+1} \tLoss: {loss.item() / len(data):.4f}')
    print(f'====> Epoch: {epoch+1} Average loss: {train_loss / len(train_loader.dataset):.4f}')

Important Notes

Training longer improves the quality of generated data.

VAE learns a smooth space to create new data by sampling latent points.

Use GPU if available for faster training.

Summary

Variational Autoencoders learn to compress and generate data using a probabilistic approach.

They use an encoder to get mean and variance, then sample a point to decode back.

Loss combines how well data is rebuilt and how close the latent space is to normal.

Practice

1. What is the main purpose of the encoder in a Variational Autoencoder (VAE)?

easy

A. To calculate the loss function

B. To produce the mean and variance for the latent space distribution

C. To generate new data samples from noise

D. To directly reconstruct the input data

Solution

1. Step 1: Understand the role of the encoder in VAE

   The encoder processes input data and outputs parameters of a probability distribution, typically mean and variance, representing the latent space.

2. Step 2: Differentiate encoder from decoder and loss

   The decoder reconstructs data, and loss measures reconstruction and latent space closeness. Encoder specifically outputs mean and variance.

3. Final Answer:

   To produce the mean and variance for the latent space distribution -> Option B

4. Quick Check:

   Encoder outputs mean and variance = D [OK]

Hint: Encoder outputs distribution parameters, not data directly [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder reconstructs data
• Assuming encoder calculates loss

2. Which PyTorch function is typically used to sample from the latent space in a VAE using the reparameterization trick?

easy

A. torch.randn()

B. torch.sigmoid()

C. torch.softmax()

D. torch.exp()

Solution

1. Step 1: Identify sampling method in VAE

   The reparameterization trick samples from a normal distribution using random noise, typically generated by torch.randn() in PyTorch.

2. Step 2: Understand other functions' roles

   torch.sigmoid() and torch.softmax() are activation functions, torch.exp() computes exponentials, none sample noise.

3. Final Answer:

   torch.randn() -> Option A

4. Quick Check:

   Sampling noise uses torch.randn() = A [OK]

Hint: Sampling noise uses torch.randn(), not activations [OK]

Common Mistakes:

• Using torch.sigmoid() to sample noise
• Confusing softmax with sampling
• Using torch.exp() incorrectly for sampling

3. Given the following PyTorch code snippet for the reparameterization trick in a VAE, what is the shape of the output z if mu and log_var have shape (batch_size, latent_dim)?

std = torch.exp(0.5 * log_var)
epsilon = torch.randn_like(std)
z = mu + epsilon * std

medium

A. (batch_size, latent_dim)

B. (latent_dim, batch_size)

C. (batch_size, 1)

D. (1, latent_dim)

Solution

1. Step 1: Analyze shapes of mu and log_var

   Both mu and log_var have shape (batch_size, latent_dim), so std and epsilon will have the same shape due to torch.exp and torch.randn_like.

2. Step 2: Calculate shape of z

   Since z = mu + epsilon * std, all operands have shape (batch_size, latent_dim), so z also has shape (batch_size, latent_dim).

3. Final Answer:

   (batch_size, latent_dim) -> Option A

4. Quick Check:

   Output shape matches input batch and latent dims = A [OK]

Hint: Output shape matches mu and log_var shapes [OK]

Common Mistakes:

• Swapping batch_size and latent_dim
• Assuming output is reduced to (batch_size, 1)
• Confusing shape broadcasting rules

4. Identify the error in this VAE loss function implementation in PyTorch:

def vae_loss(recon_x, x, mu, log_var):
    BCE = nn.functional.binary_cross_entropy(recon_x, x)
    KLD = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
    return BCE + KLD

medium

A. Incorrect sign in KLD calculation

B. Using torch.sum instead of torch.mean for KLD

C. Missing reduction='sum' in binary_cross_entropy causing scale mismatch

D. Not detaching mu and log_var before loss

Solution

1. Step 1: Check binary_cross_entropy usage

   By default, binary_cross_entropy returns mean loss per element. Summing KLD but not BCE causes scale mismatch.

2. Step 2: Fix BCE reduction

   Adding reduction='sum' to BCE matches scale with KLD, ensuring proper combined loss.

3. Final Answer:

   Missing reduction='sum' in binary_cross_entropy causing scale mismatch -> Option C

4. Quick Check:

   BCE needs reduction='sum' to match KLD scale = B [OK]

Hint: Match loss scales by using reduction='sum' in BCE [OK]

Common Mistakes:

• Ignoring BCE reduction parameter
• Changing KLD sign incorrectly
• Detaching mu/log_var unnecessarily

5. You want to train a VAE on images of size 28x28 pixels with a latent dimension of 20. Which of the following model architectures is most appropriate for the encoder part in PyTorch?

hard

A. A recurrent neural network processing images as sequences, outputting 20 values

B. A single linear layer flattening input and outputting 20 values directly

C. A fully connected network with no nonlinearities outputting 28x28 values

D. A convolutional network with Conv2d layers reducing spatial size, ending with linear layers outputting mean and log variance vectors of size 20

Solution

1. Step 1: Consider input data type and size

   Images of 28x28 pixels are spatial data; convolutional layers efficiently extract features and reduce spatial dimensions.

2. Step 2: Match output to latent dimension

   The encoder must output two vectors (mean and log variance) each of size 20, so linear layers after convolutions are suitable.

3. Step 3: Evaluate other options

   Single linear layer ignores spatial structure; RNNs are not typical for images; fully connected without nonlinearities won't learn complex features.

4. Final Answer:

   A convolutional network with Conv2d layers reducing spatial size, ending with linear layers outputting mean and log variance vectors of size 20 -> Option D

5. Quick Check:

   Conv layers + linear output for mean/log_var = C [OK]

Hint: Use conv layers for images, linear for latent params [OK]

Common Mistakes:

• Using only linear layers ignoring image structure
• Choosing RNNs for image encoding
• Omitting nonlinearities in encoder