Variational Autoencoder in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Variational Autoencoder

Which metric matters for Variational Autoencoder and WHY

A Variational Autoencoder (VAE) is a model that learns to compress and recreate images or data. The key metric to check is the reconstruction loss, which measures how close the output image is to the original input. This tells us how well the model can recreate data.

Another important metric is the Kullback-Leibler (KL) divergence. It measures how close the learned data distribution is to a normal distribution. This helps the model learn a smooth and meaningful latent space for generating new images.

In summary, we want low reconstruction loss (good image quality) and a balanced KL divergence (good latent space structure).

Confusion matrix or equivalent visualization

VAEs are unsupervised models, so they don't use confusion matrices like classifiers. Instead, we look at these values during training:

Epoch | Reconstruction Loss | KL Divergence | Total Loss
-----------------------------------------------
  1   |       120.5         |     15.3      |   135.8
  2   |       110.2         |     14.8      |   125.0
  3   |       105.0         |     14.5      |   119.5
  ... 
    

Lower reconstruction loss means better image recreation. KL divergence should not be too high or too low to keep the latent space useful.

Precision vs Recall tradeoff (or equivalent) with concrete examples

For VAEs, the main tradeoff is between reconstruction quality and latent space regularization.

• If the model focuses too much on reconstruction loss, it may memorize training images and produce blurry or less diverse outputs.
• If the model focuses too much on KL divergence, it may produce diverse but poor quality images that don't match the input well.

Example: Imagine a photo app that compresses images. If it compresses too much (high KL), photos look blurry. If it compresses too little (low KL), the app uses too much space and can't create new styles.

What "good" vs "bad" metric values look like for this use case

Good VAE metrics:

• Reconstruction loss steadily decreases and stabilizes at a low value (e.g., below 100 for image pixel MSE).
• KL divergence is moderate, not zero and not too large (e.g., between 10 and 20), indicating a balanced latent space.
• Generated images look clear and diverse.

Bad VAE metrics:

• Reconstruction loss stays high or fluctuates wildly, meaning poor image recreation.
• KL divergence is near zero (posterior collapse), meaning the model ignores the latent space.
• KL divergence is very high, causing poor reconstructions and unstable training.
• Generated images are blurry, repetitive, or nonsensical.

Metrics pitfalls

• Posterior collapse: KL divergence goes to zero, meaning the model ignores the latent space and acts like a normal autoencoder. This reduces generative power.
• Overfitting: Very low reconstruction loss on training but poor results on new data means the model memorized training images.
• Ignoring KL divergence: Focusing only on reconstruction loss can cause poor latent space structure and bad generation.
• Using only pixel-wise loss: Pixel loss may not capture perceptual quality well; sometimes perceptual or adversarial losses help.

Self-check question

Your VAE model has a reconstruction loss of 50 (low) but KL divergence near zero. Is this good?

Answer: No, this means the model ignores the latent space (posterior collapse). It may reconstruct well but cannot generate diverse new images. You should adjust training to increase KL divergence.

Key Result

For VAEs, balance low reconstruction loss with moderate KL divergence to ensure good image quality and meaningful latent space.