GAN for image generation in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - GAN for image generation

Which metric matters for GAN image generation and WHY

For GANs (Generative Adversarial Networks) that create images, we want to know how real or good the generated images look. Common metrics include:

• Inception Score (IS): Measures if generated images are clear and diverse by using a pre-trained image classifier.
• Fréchet Inception Distance (FID): Compares statistics of real and generated images to see how close they are.
• Precision and Recall for GANs: Precision checks if generated images are realistic, recall checks if the GAN creates diverse images covering the real data variety.

These metrics help us understand if the GAN is making images that look real and varied, which is the goal.

Confusion matrix or equivalent visualization

GANs do not use a traditional confusion matrix because they generate images, not classify them. Instead, we use visual and statistical comparisons:

Real images stats: mean = μ_r, covariance = Σ_r
Generated images stats: mean = μ_g, covariance = Σ_g

FID = ||μ_r - μ_g||^2 + Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2))

Lower FID means generated images are closer to real images.
    

We also look at sample images side-by-side to judge quality and diversity.

Precision vs Recall tradeoff with concrete examples

In GANs:

• High Precision, Low Recall: Generated images look very real but lack variety. For example, a GAN that only creates one type of cat image perfectly but no other cats.
• High Recall, Low Precision: GAN creates many different images but some look fake or blurry. For example, many different cats but some look strange or unrealistic.

Good GANs balance both: images look real (high precision) and cover many types (high recall).

What "good" vs "bad" metric values look like for GAN image generation

• Inception Score (IS): Higher is better. Good GANs have IS around 8-10 on datasets like CIFAR-10. Bad GANs have IS close to 1 (random noise).
• Fréchet Inception Distance (FID): Lower is better. Good GANs have FID below 50 on common datasets; top GANs can get below 10. Bad GANs have FID above 100.
• Precision and Recall: Good GANs have both precision and recall above 0.7. Bad GANs have one or both below 0.3.

Visual inspection is also key: good GAN images look sharp and varied; bad ones look blurry or repetitive.

Common pitfalls in GAN metrics

• Overfitting: GAN memorizes training images, so metrics look good but new images are not truly generated.
• Mode collapse: GAN generates limited types of images, hurting recall but sometimes precision looks good.
• Misleading IS: Inception Score can be high if images are clear but not diverse.
• Data leakage: Using test images in training can falsely improve metrics.
• Ignoring visual quality: Metrics alone can miss artifacts or unnatural details.

Self-check question

Your GAN has an Inception Score of 9.5 but a Fréchet Inception Distance of 120. Is it good for production? Why or why not?

Answer: No, it is not good. The high IS means images look clear, but the very high FID shows generated images are far from real ones in distribution. This likely means poor diversity or unrealistic details. Both metrics must be good for reliable GAN performance.

Key Result

For GAN image generation, balance between image realism (precision) and diversity (recall) measured by FID and Inception Score is key.