Why generative models create data in PyTorch - Why Metrics Matter

Metrics & Evaluation - Why generative models create data

Which metric matters for this concept and WHY

For generative models that create data, we want to know how well the model mimics real data. Key metrics include Likelihood (how probable the model thinks real data is), Frechet Inception Distance (FID) (how close generated images are to real ones), and Inception Score (IS). These metrics tell us if the generated data looks real and diverse. Likelihood measures fit to data, while FID and IS measure quality and variety of generated samples.

Confusion matrix or equivalent visualization (ASCII)

    Real Data vs Generated Data Similarity (Conceptual)

          | Real Data | Generated Data
    ------|-----------|----------------
    Real  |    TP     |       FN       
    Data  | (real data| (real data     
          | classified| classified as  
          | as real)  | generated)     
    ------|-----------|----------------
    Fake  |    FP     |       TN       
    Data  | (generated| (generated data 
          | data      | classified as  
          | classified| fake)          
          | as real)  |                

    Note: This is a conceptual analogy. Generative models don't use confusion matrices directly but metrics like FID capture similar ideas of similarity.
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

In generative models, precision means how many generated samples look real (quality), and recall means how much of the real data variety the model covers (diversity).

Example: A model that creates only one perfect image has high precision but low recall (no variety). A model that creates many varied images but some look fake has high recall but lower precision.

Good generative models balance both: they create many different images that all look real.

What "good" vs "bad" metric values look like for this use case

• Likelihood: Higher is better. Good models assign high probability to real data.
• FID: Lower is better. Good models have FID close to 0, meaning generated data is very similar to real data.
• Inception Score: Higher is better. Good models produce clear, diverse images.

Bad example: High FID (e.g., 100+) means generated data looks very different from real data. Low Inception Score means images are blurry or repetitive.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Mode collapse: Model generates limited variety (low recall) but metrics like likelihood may still look okay.
• Overfitting: Model memorizes training data, so generated samples look good but model fails to generalize.
• Data leakage: If test data leaks into training, metrics falsely improve.
• Misleading likelihood: Some models have intractable likelihood, so surrogate metrics are needed.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

This question is from classification but helps understand tradeoffs. A model with 98% accuracy but only 12% recall on fraud misses most fraud cases. For fraud detection, recall is critical because missing fraud is costly. So, despite high accuracy, this model is not good for production.

Similarly, for generative models, high quality (precision) alone is not enough if diversity (recall) is very low.

Key Result

Generative models need metrics that measure both quality (precision) and diversity (recall) of created data to ensure realistic and varied outputs.

Practice

1. Why do generative models create new data in machine learning?

easy

A. To speed up training by skipping data

B. To delete existing data from the dataset

C. To only classify data into categories

D. To mimic real examples and generate similar data

Solution

1. Step 1: Understand the purpose of generative models

   Generative models learn patterns from real data to create new, similar data.

2. Step 2: Identify the correct purpose from options

   Only To mimic real examples and generate similar data describes creating new data that mimics real examples, which is the main goal.

3. Final Answer:

   To mimic real examples and generate similar data -> Option D

4. Quick Check:

   Generative models create data = mimic real examples [OK]

Hint: Generative models create data similar to real examples [OK]

Common Mistakes:

• Confusing generative models with classifiers
• Thinking generative models delete data
• Assuming generative models speed up training by skipping data

2. Which PyTorch code snippet correctly initializes a simple generative model layer?

easy

A. torch.nn.Linear(100, 256)

B. torch.nn.Conv2d(3, 64, kernel_size=3)

C. torch.optim.SGD(model.parameters(), lr=0.01)

D. torch.nn.CrossEntropyLoss()

Solution

1. Step 1: Identify generative model layer types

   Generative models often use linear layers to transform random input vectors into data representations.

2. Step 2: Match code to layer initialization

   torch.nn.Linear(100, 256) correctly initializes a linear layer with input size 100 and output size 256, suitable for generative models.

3. Final Answer:

   torch.nn.Linear(100, 256) -> Option A

4. Quick Check:

   Linear layer init = torch.nn.Linear(100, 256) [OK]

Hint: Linear layers often start generative models [OK]

Common Mistakes:

• Choosing optimizer or loss function as layer
• Confusing convolutional layers with simple linear layers
• Using loss functions as model layers

3. Given this PyTorch code, what is the shape of the generated data tensor?

import torch
noise = torch.randn(16, 100)
generator = torch.nn.Linear(100, 784)
output = generator(noise)

medium

A. (16, 784)

B. (16, 100)

C. (100, 784)

D. (784, 16)

Solution

1. Step 1: Understand input and output shapes

   The noise tensor has shape (16, 100), meaning 16 samples each with 100 features. The linear layer maps 100 features to 784 features.

2. Step 2: Calculate output shape after linear layer

   Applying the linear layer to noise results in shape (16, 784), preserving batch size and changing feature size.

3. Final Answer:

   (16, 784) -> Option A

4. Quick Check:

   Output shape = (batch_size, output_features) = (16, 784) [OK]

Hint: Output shape matches batch size and output features [OK]

Common Mistakes:

• Mixing batch size and feature dimensions
• Confusing input and output shapes
• Transposing shapes incorrectly

4. Find the error in this PyTorch code snippet for generating data:

import torch
noise = torch.randn(32, 50)
generator = torch.nn.Linear(100, 784)
output = generator(noise)

medium

A. Output tensor shape is correct

B. Generator layer should be torch.nn.Conv2d

C. Noise tensor has wrong shape for the generator input

D. Missing optimizer initialization

Solution

1. Step 1: Check input shape compatibility

   The generator expects input with 100 features, but noise has 50 features per sample, causing a shape mismatch error.

2. Step 2: Identify the error cause

   Because noise shape (32, 50) does not match expected input (batch_size, 100), the code will fail at runtime.

3. Final Answer:

   Noise tensor has wrong shape for the generator input -> Option C

4. Quick Check:

   Input shape mismatch = noise shape wrong [OK]

Hint: Input features must match layer input size [OK]

Common Mistakes:

• Ignoring input-output shape mismatch
• Thinking convolutional layers are always needed
• Confusing optimizer with model layers

5. Why do generative models use random noise as input when creating new data?

hard

A. To speed up training by skipping data

B. To introduce variety and generate diverse outputs

C. To reduce the size of the model

D. To make the model deterministic and fixed

Solution

1. Step 1: Understand the role of random noise in generative models

   Random noise acts as a seed to produce different outputs, allowing the model to generate varied data samples.

2. Step 2: Match the purpose with the correct option

   To introduce variety and generate diverse outputs correctly states that noise introduces variety and diversity in generated data.

3. Final Answer:

   To introduce variety and generate diverse outputs -> Option B

4. Quick Check:

   Noise input = variety in outputs [OK]

Hint: Noise input creates diverse generated data [OK]

Common Mistakes:

• Thinking noise speeds up training
• Assuming noise reduces model size
• Believing noise makes output fixed