GAN for image generation in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - GAN for image generation

Which metric matters for GAN image generation and WHY

For GANs (Generative Adversarial Networks) that create images, we want to know how real or good the generated images look. Common metrics include:

• Inception Score (IS): Measures if generated images are clear and diverse by using a pre-trained image classifier.
• Fréchet Inception Distance (FID): Compares statistics of real and generated images to see how close they are.
• Precision and Recall for GANs: Precision checks if generated images are realistic, recall checks if the GAN creates diverse images covering the real data variety.

These metrics help us understand if the GAN is making images that look real and varied, which is the goal.

Confusion matrix or equivalent visualization

GANs do not use a traditional confusion matrix because they generate images, not classify them. Instead, we use visual and statistical comparisons:

Real images stats: mean = μ_r, covariance = Σ_r
Generated images stats: mean = μ_g, covariance = Σ_g

FID = ||μ_r - μ_g||^2 + Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2))

Lower FID means generated images are closer to real images.
    

We also look at sample images side-by-side to judge quality and diversity.

Precision vs Recall tradeoff with concrete examples

In GANs:

• High Precision, Low Recall: Generated images look very real but lack variety. For example, a GAN that only creates one type of cat image perfectly but no other cats.
• High Recall, Low Precision: GAN creates many different images but some look fake or blurry. For example, many different cats but some look strange or unrealistic.

Good GANs balance both: images look real (high precision) and cover many types (high recall).

What "good" vs "bad" metric values look like for GAN image generation

• Inception Score (IS): Higher is better. Good GANs have IS around 8-10 on datasets like CIFAR-10. Bad GANs have IS close to 1 (random noise).
• Fréchet Inception Distance (FID): Lower is better. Good GANs have FID below 50 on common datasets; top GANs can get below 10. Bad GANs have FID above 100.
• Precision and Recall: Good GANs have both precision and recall above 0.7. Bad GANs have one or both below 0.3.

Visual inspection is also key: good GAN images look sharp and varied; bad ones look blurry or repetitive.

Common pitfalls in GAN metrics

• Overfitting: GAN memorizes training images, so metrics look good but new images are not truly generated.
• Mode collapse: GAN generates limited types of images, hurting recall but sometimes precision looks good.
• Misleading IS: Inception Score can be high if images are clear but not diverse.
• Data leakage: Using test images in training can falsely improve metrics.
• Ignoring visual quality: Metrics alone can miss artifacts or unnatural details.

Self-check question

Your GAN has an Inception Score of 9.5 but a Fréchet Inception Distance of 120. Is it good for production? Why or why not?

Answer: No, it is not good. The high IS means images look clear, but the very high FID shows generated images are far from real ones in distribution. This likely means poor diversity or unrealistic details. Both metrics must be good for reliable GAN performance.

Key Result

For GAN image generation, balance between image realism (precision) and diversity (recall) measured by FID and Inception Score is key.

Practice

1. What are the two main parts of a GAN used for image generation?

easy

A. Convolution and Pooling

B. Encoder and Decoder

C. Input Layer and Output Layer

D. Generator and Discriminator

Solution

1. Step 1: Understand GAN components

   A GAN consists of two parts: a generator that creates images and a discriminator that evaluates them.

2. Step 2: Match parts to options

   Only Generator and Discriminator correctly names these two parts as Generator and Discriminator.

3. Final Answer:

   Generator and Discriminator -> Option D

4. Quick Check:

   GAN parts = Generator and Discriminator [OK]

Hint: Remember GAN = Generator + Discriminator [OK]

Common Mistakes:

• Confusing GAN parts with autoencoder parts
• Thinking GAN uses Encoder and Decoder
• Mixing up layers with GAN components

2. Which of the following is the correct way to define the generator model in PyTorch for a GAN?

easy

A. generator = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 100), nn.Softmax())

B. generator = nn.Sequential(nn.Conv2d(1, 64, 3), nn.Sigmoid())

C. generator = nn.Sequential(nn.Linear(100, 256), nn.ReLU(), nn.Linear(256, 784), nn.Tanh())

D. generator = nn.Sequential(nn.ConvTranspose2d(100, 1, 3), nn.ReLU())

Solution

1. Step 1: Identify generator input and output sizes

   The generator takes a noise vector (size 100) and outputs an image vector (size 784 for 28x28 images).

2. Step 2: Check model layers and activations

   generator = nn.Sequential(nn.Linear(100, 256), nn.ReLU(), nn.Linear(256, 784), nn.Tanh()) correctly uses Linear layers from 100 to 256 to 784 with ReLU and Tanh activations, common in GAN generators.

3. Final Answer:

   generator = nn.Sequential(nn.Linear(100, 256), nn.ReLU(), nn.Linear(256, 784), nn.Tanh()) -> Option C

4. Quick Check:

   Generator input=100, output=784 with ReLU and Tanh [OK]

Hint: Generator maps noise to image size with Linear + Tanh [OK]

Common Mistakes:

• Using Conv2d instead of Linear for generator input
• Reversing input-output sizes
• Using Softmax instead of Tanh at output

3. Given this PyTorch snippet for a GAN training step, what will be the shape of the generated images tensor?

noise = torch.randn(16, 100)
generated_images = generator(noise)

medium

A. (16, 784)

B. (16, 100)

C. (100, 16)

D. (784, 16)

Solution

1. Step 1: Understand input noise shape

   The noise tensor has shape (16, 100), meaning batch size 16 and noise vector length 100.

2. Step 2: Check generator output shape

   The generator maps each noise vector of size 100 to an image vector of size 784, so output shape is (16, 784).

3. Final Answer:

   (16, 784) -> Option A

4. Quick Check:

   Batch size 16, output image size 784 [OK]

Hint: Output shape matches batch size and image vector size [OK]

Common Mistakes:

• Confusing input noise shape with output shape
• Swapping batch size and feature dimensions
• Assuming output shape is (16, 100)

4. You have this error during GAN training: RuntimeError: size mismatch in Linear layer. What is the most likely cause?

medium

A. The input noise vector size does not match the first Linear layer input size

B. The discriminator output is not a probability

C. The generator uses ReLU instead of Tanh

D. The batch size is too large

Solution

1. Step 1: Understand size mismatch error

   This error usually means the input tensor size does not match the expected input size of a Linear layer.

2. Step 2: Identify likely cause in GAN

   If the noise vector size does not match the first Linear layer's input features in the generator, this error occurs.

3. Final Answer:

   The input noise vector size does not match the first Linear layer input size -> Option A

4. Quick Check:

   Size mismatch = input noise size error [OK]

Hint: Check noise vector size matches Linear layer input [OK]

Common Mistakes:

• Blaming activation functions for size errors
• Thinking batch size causes size mismatch
• Ignoring input dimension mismatch

5. You want to improve the quality of images generated by your GAN. Which of these strategies is most effective?

hard

A. Increase the size of the noise vector and train longer

B. Use a deeper generator and discriminator with batch normalization

C. Remove the discriminator and train only the generator

D. Use random labels instead of real/fake labels for training

Solution

1. Step 1: Consider model architecture improvements

   Deeper networks with batch normalization help stabilize training and improve image quality.

2. Step 2: Evaluate other options

   Increasing noise size alone or removing discriminator harms quality; random labels confuse training.

3. Final Answer:

   Use a deeper generator and discriminator with batch normalization -> Option B

4. Quick Check:

   Better architecture + batch norm = improved GAN images [OK]

Hint: Better models + batch norm improve GAN image quality [OK]

Common Mistakes:

• Thinking bigger noise vector alone improves quality
• Removing discriminator to simplify training
• Using random labels instead of correct ones