Autoencoder architecture in PyTorch

Introduction

An autoencoder learns to copy its input to its output by compressing data into a smaller form and then reconstructing it. This helps find important features in data without labels.

To reduce the size of images or data for easier storage or faster processing.

To remove noise from pictures or signals by learning clean versions.

To learn useful features from data when you don't have labels.

To generate new data similar to the input by sampling from the compressed form.

Syntax

PyTorch

class Autoencoder(nn.Module):
    def __init__(self, input_size, hidden_size, code_size):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, code_size),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.Linear(code_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, input_size),
            nn.Sigmoid()
        )
    def forward(self, x):
        code = self.encoder(x)
        reconstructed = self.decoder(code)
        return reconstructed

The encoder compresses the input into a smaller code.

The decoder tries to rebuild the original input from the code.

Examples

A very simple autoencoder for 28x28 images flattened to 784 features.

PyTorch

class SimpleAutoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 32)
        self.decoder = nn.Linear(32, 784)
    def forward(self, x):
        code = torch.relu(self.encoder(x))
        reconstructed = torch.sigmoid(self.decoder(code))
        return reconstructed

A deeper autoencoder with multiple layers compressing to 3 features.

PyTorch

class DeepAutoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 12),
            nn.ReLU(),
            nn.Linear(12, 3)
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(),
            nn.Linear(12, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 784),
            nn.Sigmoid()
        )
    def forward(self, x):
        code = self.encoder(x)
        reconstructed = self.decoder(code)
        return reconstructed

Sample Model

This program trains a simple autoencoder on MNIST digits for one batch and prints the loss. The loss shows how well the model reconstructs the input.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Define Autoencoder
class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(28*28, 64),
            nn.ReLU(),
            nn.Linear(64, 12),
            nn.ReLU(),
            nn.Linear(12, 3)
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(),
            nn.Linear(12, 64),
            nn.ReLU(),
            nn.Linear(64, 28*28),
            nn.Sigmoid()
        )
    def forward(self, x):
        code = self.encoder(x)
        reconstructed = self.decoder(code)
        return reconstructed

# Load MNIST dataset
transform = transforms.Compose([transforms.ToTensor(), transforms.Lambda(lambda x: x.view(-1))])
mnist_train = datasets.MNIST(root='.', train=True, download=True, transform=transform)
train_loader = DataLoader(mnist_train, batch_size=128, shuffle=True)

# Initialize model, loss, optimizer
model = Autoencoder()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train for 1 epoch
model.train()
for batch_idx, (data, _) in enumerate(train_loader):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, data)
    loss.backward()
    optimizer.step()
    if batch_idx == 0:
        print(f'Batch {batch_idx} Loss: {loss.item():.4f}')
        break

Important Notes

Use a sigmoid activation at the output if input data is normalized between 0 and 1.

The size of the code (bottleneck) controls how much the data is compressed.

Training longer and with more data improves reconstruction quality.

Summary

An autoencoder learns to compress and then reconstruct data.

It has two parts: encoder (compress) and decoder (rebuild).

It helps find important features without needing labels.

Practice

1. What is the main purpose of the encoder part in an autoencoder?

easy

A. To generate new random data samples

B. To reconstruct the original input from compressed data

C. To classify the input data into categories

D. To compress the input data into a smaller representation

Solution

1. Step 1: Understand the role of encoder

   The encoder compresses input data into a smaller form called latent space or code.

2. Step 2: Differentiate encoder from decoder

   The decoder rebuilds the original data from this compressed form, not the encoder.

3. Final Answer:

   To compress the input data into a smaller representation -> Option D

4. Quick Check:

   Encoder compresses data = A [OK]

Hint: Encoder compresses, decoder reconstructs [OK]

Common Mistakes:

• Confusing encoder with decoder
• Thinking encoder classifies data
• Assuming encoder generates new data

2. Which of the following is the correct way to define a simple encoder layer in PyTorch?

easy

A. self.encoder = nn.Linear(input_dim, hidden_dim)

B. self.encoder = nn.Conv2d(hidden_dim, input_dim)

C. self.encoder = nn.ReLU(input_dim, hidden_dim)

D. self.encoder = nn.MaxPool2d(input_dim, hidden_dim)

Solution

1. Step 1: Identify correct PyTorch layer for encoding

   Encoding usually uses a linear layer to reduce dimensions: nn.Linear(input_dim, hidden_dim).

2. Step 2: Check other options for correctness

   Conv2d requires 4D input and different parameters; ReLU is an activation, not a layer; MaxPool2d is for downsampling images, not linear encoding.

3. Final Answer:

   self.encoder = nn.Linear(input_dim, hidden_dim) -> Option A

4. Quick Check:

   Linear layer for encoding = B [OK]

Hint: Use nn.Linear for encoder layers in simple autoencoders [OK]

Common Mistakes:

• Using activation functions as layers
• Confusing convolution with linear layers
• Wrong parameter order in layers

3. Given this PyTorch autoencoder snippet, what is the shape of the output tensor after encoding if input shape is (batch_size=10, features=784)?

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 32)
    def forward(self, x):
        x = self.encoder(x)
        return x
model = Autoencoder()
input_tensor = torch.randn(10, 784)
output = model(input_tensor)

medium

A. (784, 32)

B. (10, 32)

C. (10, 784)

D. (32, 784)

Solution

1. Step 1: Understand input and encoder layer

   Input tensor shape is (10, 784). Encoder is nn.Linear(784, 32), so it maps 784 features to 32 features per sample.

2. Step 2: Calculate output shape after encoder

   Output shape keeps batch size 10, features become 32, so output shape is (10, 32).

3. Final Answer:

   (10, 32) -> Option B

4. Quick Check:

   Batch size unchanged, features to 32 = D [OK]

Hint: Linear layer output shape = (batch_size, output_features) [OK]

Common Mistakes:

• Swapping batch size and feature dimensions
• Confusing input and output feature sizes
• Assuming shape changes batch size

4. Identify the error in this autoencoder decoder definition:

class Decoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.decoder = nn.Linear(32, 784)
    def forward(self, x):
        x = self.decoder(x)
        x = nn.ReLU(x)
        return x

medium

A. Missing activation function after linear layer

B. The decoder input and output dimensions are swapped

C. nn.ReLU is used incorrectly as a function instead of a layer

D. The forward method should return x before applying ReLU

Solution

1. Step 1: Check usage of nn.ReLU

   nn.ReLU is a class and must be instantiated before use, e.g. nn.ReLU()(x), not nn.ReLU(x).

2. Step 2: Verify decoder dimensions and flow

   Decoder maps 32 to 784 correctly, and activation after linear is common, so no issue there.

3. Final Answer:

   nn.ReLU is used incorrectly as a function instead of a layer -> Option C

4. Quick Check:

   Activation must be instantiated = C [OK]

Hint: Instantiate nn.ReLU() before calling it on tensor [OK]

Common Mistakes:

• Calling nn.ReLU as a function directly
• Swapping input/output dims in Linear
• Forgetting to apply activation

5. You want to build an autoencoder that compresses 1000-dimensional input to 10 dimensions and then reconstructs it. Which architecture below correctly defines encoder and decoder layers in PyTorch?

hard

A. ""class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(1000, 100),
nn.ReLU(),
nn.Linear(100, 10)
)
self.decoder = nn.Sequential(
nn.Linear(10, 100),
nn.ReLU(),
nn.Linear(100, 1000),
nn.Sigmoid()
)
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x"""

B. ""class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(10, 100),
nn.ReLU(),
nn.Linear(100, 1000)
)
self.decoder = nn.Sequential(
nn.Linear(1000, 100),
nn.ReLU(),
nn.Linear(100, 10)
)
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x"""

C. ""class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(1000, 20),
nn.ReLU()
)
self.decoder = nn.Sequential(
nn.Linear(20, 1000),
nn.Sigmoid()
)
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x"""

D. ""class Autoencoder(nn.Module):
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(1000, 500),
nn.ReLU(),
nn.Linear(500, 50)
)
self.decoder = nn.Sequential(
nn.Linear(50, 500),
nn.ReLU(),
nn.Linear(500, 1000),
nn.Sigmoid()
)
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x"""

Solution

1. Step 1: Check encoder input and output sizes

   Input is 1000, compressed to 10. ""class Autoencoder(nn.Module): def __init__(self): super().__init__() self.encoder = nn.Sequential( nn.Linear(1000, 100), nn.ReLU(), nn.Linear(100, 10) ) self.decoder = nn.Sequential( nn.Linear(10, 100), nn.ReLU(), nn.Linear(100, 1000), nn.Sigmoid() ) def forward(self, x): x = self.encoder(x) x = self.decoder(x) return x""" correctly reduces 1000 -> 100 -> 10. Others have wrong input/output sizes.

2. Step 2: Check decoder mirrors encoder

   Decoder in A expands 10 -> 100 -> 1000, matching encoder compression steps. ""class Autoencoder(nn.Module): def __init__(self): super().__init__() self.encoder = nn.Sequential( nn.Linear(1000, 500), nn.ReLU(), nn.Linear(500, 50) ) self.decoder = nn.Sequential( nn.Linear(50, 500), nn.ReLU(), nn.Linear(500, 1000), nn.Sigmoid() ) def forward(self, x): x = self.encoder(x) x = self.decoder(x) return x""" has wrong final layer input size (50 to 1000 is correct, but encoder compresses to 50, not 10). Options B and C have reversed or incorrect sizes.

3. Final Answer:

   Option A correctly defines encoder and decoder layers for 1000 to 10 compression and reconstruction -> Option A

4. Quick Check:

   Encoder compresses 1000 to 10, decoder reverses = A [OK]

Hint: Encoder reduces dims stepwise; decoder reverses steps [OK]

Common Mistakes:

• Swapping input/output sizes in layers
• Skipping intermediate compression layers
• Mismatching encoder and decoder sizes