Autoencoder for images in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Autoencoder for images

Which metric matters for Autoencoder for images and WHY

For autoencoders working with images, the main goal is to recreate the input image as closely as possible. So, we use reconstruction error metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE). These numbers tell us how different the output image is from the input image. A smaller error means the autoencoder is better at capturing important details.

Sometimes, if the autoencoder is used for anomaly detection, we look at the reconstruction error to spot unusual images. Higher error means the image is different from what the model learned.

Confusion matrix or equivalent visualization

Autoencoders don't usually use confusion matrices because they don't classify images. Instead, we look at error values. Here is an example of reconstruction error values for 5 images:

    Image ID | Reconstruction Error (MSE)
    ---------|--------------------------
    1        | 0.002
    2        | 0.0015
    3        | 0.005
    4        | 0.0008
    5        | 0.007
    

Lower values mean better reconstruction. If used for anomaly detection, images with error above a threshold (like 0.004) might be flagged as anomalies.

Precision vs Recall tradeoff with concrete examples

When autoencoders detect anomalies, we set a threshold on reconstruction error. This creates a tradeoff:

• High threshold: Few images flagged as anomalies. This means high precision (most flagged are true anomalies) but low recall (many anomalies missed).
• Low threshold: Many images flagged. This means high recall (catch most anomalies) but low precision (many normal images wrongly flagged).

Example: In a factory, missing a defect (low recall) can be costly, so we prefer higher recall even if precision drops.

What "good" vs "bad" metric values look like for this use case

Good: Low reconstruction error (e.g., MSE < 0.001) on normal images means the autoencoder learns well. For anomaly detection, a clear gap between errors of normal and abnormal images is ideal.

Bad: High reconstruction error on normal images means the model is not learning patterns well. Overlapping error values for normal and abnormal images make it hard to detect anomalies.

Metrics pitfalls

• Ignoring reconstruction error scale: Different image sizes or pixel ranges affect error values. Always normalize or compare errors consistently.
• Overfitting: Very low error on training images but high error on new images means the model memorized instead of learned.
• Data leakage: If test images are too similar to training, error looks low but model may fail on real new data.
• Using accuracy: Accuracy is not meaningful here because autoencoders don't classify.

Self-check question

Your autoencoder has a reconstruction error of 0.0005 on training images but 0.01 on new images. Is it good for production? Why or why not?

Answer: No, this shows overfitting. The model learned training images too well but cannot generalize to new images. It needs better training or regularization.

Key Result

Reconstruction error (like MSE) is key to measure how well an autoencoder recreates images; low error means good learning and useful anomaly detection.