Autoencoder architecture in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Model Choice

intermediate

Choose the correct autoencoder architecture

Which of the following PyTorch model architectures correctly implements a simple autoencoder with one hidden layer in the encoder and decoder?

A

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.Linear(128, 784),
            nn.Sigmoid()
        )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

B

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.Sigmoid()
        )
        self.decoder = nn.Sequential(
            nn.Linear(128, 784),
            nn.ReLU()
        )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

C

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.Tanh()
        )
        self.decoder = nn.Sequential(
            nn.Linear(128, 784),
            nn.Tanh()
        )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

D

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.Linear(128, 784),
            nn.ReLU()
        )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

💡 Hint

Remember the decoder output activation should be sigmoid for normalized input images.

Option A uses ReLU in encoder and Sigmoid in decoder output which is standard for image reconstruction between 0 and 1. Option A reverses activations causing poor output range. Option A uses Tanh which outputs between -1 and 1, not suitable for normalized images. Option A uses ReLU in decoder output which can produce values outside [0,1].

❓ Predict Output

intermediate

Output shape of encoded representation

Given the following autoencoder model, what is the shape of the encoded output for an input batch of shape (64, 784)?

PyTorch

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Linear(256, 64),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.Linear(64, 256),
            nn.ReLU(),
            nn.Linear(256, 784),
            nn.Sigmoid()
        )
    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

model = Autoencoder()
input_batch = torch.randn(64, 784)
encoded_output = model.encoder(input_batch)
encoded_output.shape

A (256, 64)

B (64, 256)

C (64, 64)

D (784, 64)

💡 Hint

Look at the last Linear layer in the encoder.

The encoder ends with a Linear layer outputting 64 features. The batch size is 64, so output shape is (64, 64).

❓ Hyperparameter

advanced

Choosing the latent space size

In an autoencoder, what is the main effect of increasing the size of the latent space (the encoded representation dimension)?

A It always improves compression and reduces reconstruction error.

B It increases the model's capacity to reconstruct input but may reduce compression and generalization.

C It decreases the model's capacity and leads to underfitting.

D It has no effect on model performance or compression.

💡 Hint

Think about the trade-off between compression and reconstruction quality.

A larger latent space allows the model to store more information, improving reconstruction but reducing compression and possibly hurting generalization.

❓ Metrics

advanced

Choosing the loss function for image autoencoder

Which loss function is most appropriate for training an autoencoder on normalized grayscale images with pixel values between 0 and 1?

A Cosine similarity loss

B Cross Entropy loss

C Hinge loss

D Mean Squared Error (MSE) loss

💡 Hint

Consider the type of output and target values.

MSE loss measures the average squared difference between reconstructed and original pixel values, suitable for continuous values between 0 and 1.

🔧 Debug

expert

Identify the cause of training loss not decreasing

A user trains an autoencoder on MNIST but notices the training loss does not decrease at all. The model code is below. What is the most likely cause?

PyTorch

class Autoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.Linear(128, 784),
            nn.ReLU()
        )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = Autoencoder()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for data, _ in dataloader:
    data = data.view(data.size(0), -1)
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, data)
    loss.backward()
    optimizer.step()

A Decoder uses ReLU activation causing outputs to be >1, but input pixels are between 0 and 1, leading to poor loss optimization.

B The optimizer learning rate is too high causing divergence.

C The input data is not flattened before feeding to the model.

D The loss function is inappropriate for reconstruction tasks.

💡 Hint

Check the decoder output activation and input data range.

ReLU in decoder output can produce values greater than 1, while input pixels are between 0 and 1. This mismatch makes MSE loss hard to optimize.