Key models overview (GPT, DALL-E, Stable Diffusion) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Key models overview (GPT, DALL-E, Stable Diffusion)

Which metric matters for this concept and WHY

For models like GPT (text generation), DALL-E and Stable Diffusion (image generation), the key metrics differ because their tasks differ.

GPT: We look at perplexity to see how well the model predicts text. Lower perplexity means better predictions.

DALL-E and Stable Diffusion: We use FID (Fréchet Inception Distance) and IS (Inception Score) to measure image quality and diversity. Lower FID and higher IS mean better images.

These metrics help us know if the model creates realistic and useful outputs.

Confusion matrix or equivalent visualization (ASCII)

Since these are generative models, confusion matrices don't apply directly. Instead, we use example outputs and metric scores.

    GPT Perplexity Example:
    ----------------------
    Model predicts next word probabilities.
    Perplexity = 10 means on average the model is as uncertain as choosing among 10 words.

    DALL-E / Stable Diffusion FID Example:
    ------------------------------------
    Real images vs generated images feature comparison.
    Lower FID (e.g., 10) means generated images are close to real ones.
    Higher FID (e.g., 100) means poor quality.
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

For generative models, the tradeoff is often between quality and diversity.

Quality: How realistic and sharp the output is.

Diversity: How varied and creative the outputs are.

Example: A model that always generates the same perfect image has high quality but low diversity.

A model that generates many different images but some look blurry has high diversity but lower quality.

Good models balance both well.

What "good" vs "bad" metric values look like for this use case

GPT:

• Good perplexity: low (e.g., 10 or less on simple tasks)
• Bad perplexity: high (e.g., 100 or more means poor text prediction)

DALL-E / Stable Diffusion:

• Good FID: low (e.g., below 30 means realistic images)
• Bad FID: high (e.g., above 100 means images look fake or blurry)
• Good IS: high (e.g., above 8 means diverse and clear images)
• Bad IS: low (e.g., below 3 means poor image quality or low variety)

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Overfitting: Model memorizes training data, so metrics look great on training but poor on new inputs.
• Data leakage: If test data leaks into training, metrics falsely improve.
• Metric mismatch: Using accuracy or classification metrics on generative models is wrong.
• Ignoring diversity: Only focusing on quality can lead to repetitive outputs.
• Human evaluation needed: Metrics don't capture creativity or usefulness fully.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

This question is about fraud detection, not generative models, but it shows why metrics matter.

98% accuracy sounds good, but 12% recall means the model misses 88% of fraud cases.

This is bad because catching fraud is critical. So, despite high accuracy, the model is not good for production.

Lesson: Always check the right metrics for your task.

Key Result

Generative models require task-specific metrics like perplexity for GPT and FID/IS for image models to evaluate quality and diversity effectively.

Practice

1. Which model is mainly used to generate human-like text?

easy

A. GPT

B. DALL-E

C. Stable Diffusion

D. None of the above

Solution

1. Step 1: Understand GPT's purpose

   GPT is designed to generate and understand human-like text.

2. Step 2: Compare with other models

   DALL-E and Stable Diffusion create images, not text.

3. Final Answer:

   GPT -> Option A

4. Quick Check:

   Text generation = GPT [OK]

Hint: Text output? Think GPT first. [OK]

Common Mistakes:

• Confusing DALL-E as text model
• Thinking Stable Diffusion generates text
• Choosing 'None of the above'

2. Which of the following is the correct way to describe DALL-E's function?

easy

A. It generates text based on images.

B. It compresses images for storage.

C. It creates images from text descriptions.

D. It translates text from one language to another.

Solution

1. Step 1: Identify DALL-E's main function

   DALL-E creates images from text prompts given by users.

2. Step 2: Eliminate incorrect options

   It does not generate text, translate languages, or compress images.

3. Final Answer:

   It creates images from text descriptions. -> Option C

4. Quick Check:

   Text to image = DALL-E [OK]

Hint: DALL-E = text to image creator. [OK]

Common Mistakes:

• Thinking DALL-E generates text
• Confusing with translation models
• Assuming it compresses images

3. Given the following code snippet using a model, what type of output should you expect?

model = 'Stable Diffusion'
input_text = 'A sunny beach with palm trees'
output = model.generate(input_text)

medium

A. A photo-realistic image of a sunny beach

B. A summary of the text input

C. A written story about a beach

D. An error because Stable Diffusion cannot generate output

Solution

1. Step 1: Identify Stable Diffusion's output type

   Stable Diffusion generates images from text prompts.

2. Step 2: Match input and output

   Input is a text description; output will be an image matching that description.

3. Final Answer:

   A photo-realistic image of a sunny beach -> Option A

4. Quick Check:

   Text input + Stable Diffusion = Image output [OK]

Hint: Stable Diffusion turns words into pictures. [OK]

Common Mistakes:

• Expecting text output
• Thinking it summarizes text
• Assuming it causes an error

4. You tried to use GPT to create an image by running this code:

model = 'GPT'
input_text = 'A cat sitting on a sofa'
output = model.generate_image(input_text)

What is the main problem here?

medium

A. The input text is too short for GPT to understand.

B. GPT cannot generate images; it only generates text.

C. The method name should be generate_text, not generate_image.

D. There is no problem; the code will work fine.

Solution

1. Step 1: Understand GPT's capabilities

   GPT is designed to generate text, not images.

2. Step 2: Analyze the method call

   Calling generate_image on GPT is invalid because GPT lacks image generation ability.

3. Final Answer:

   GPT cannot generate images; it only generates text. -> Option B

4. Quick Check:

   GPT = text only, no images [OK]

Hint: GPT does text, not images. [OK]

Common Mistakes:

• Thinking GPT can create images
• Believing method name is wrong only
• Ignoring model capability limits

5. You want to build an app that lets users type a prompt to generate a story and then see an image illustrating it. Which combination of models should you use?

hard

A. Use GPT for image generation and DALL-E for text generation.

B. Use DALL-E to generate the story and GPT to create the image.

C. Use Stable Diffusion for both story and image generation.

D. Use GPT to generate the story and Stable Diffusion to create the image.

Solution

1. Step 1: Identify model roles for text and image

   GPT is best for generating human-like text stories.

2. Step 2: Identify model for image creation

   Stable Diffusion creates images from text descriptions, perfect for illustrating stories.

3. Final Answer:

   Use GPT to generate the story and Stable Diffusion to create the image. -> Option D

4. Quick Check:

   Text by GPT + Image by Stable Diffusion = App [OK]

Hint: Text with GPT, images with Stable Diffusion. [OK]

Common Mistakes:

• Swapping roles of GPT and DALL-E
• Using one model for both tasks
• Confusing image and text generation roles