Stable Diffusion overview in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Stable Diffusion overview

Which metric matters for Stable Diffusion and WHY

Stable Diffusion is a model that creates images from text. To check how well it works, we look at how realistic and relevant the images are. Common metrics include FID (Fréchet Inception Distance) which measures how close the generated images are to real ones, and CLIP score which checks if the image matches the text description. These metrics matter because they tell us if the images look good and fit the text prompt.

Confusion matrix or equivalent visualization

Stable Diffusion does not use a confusion matrix because it is a generative model, not a classifier. Instead, we use visual examples and scores like FID and CLIP to evaluate quality.

Example FID scores:
Real images vs Generated images
Lower FID = better quality

Example CLIP score:
Text prompt: "A cat sitting on a chair"
Generated image matches prompt well = High CLIP score
    

Precision vs Recall tradeoff with concrete examples

For Stable Diffusion, precision means how clear and detailed the images are, while recall means how diverse and varied the images can be for the same prompt.

If the model focuses too much on precision, images look sharp but may be very similar (low diversity). If it focuses on recall, images vary a lot but may be blurry or less accurate.

Example: For a prompt "a red apple", high precision means every apple looks very realistic and red. High recall means apples might look different shapes or styles but still red.

What "good" vs "bad" metric values look like for Stable Diffusion

Good: FID below 30 means generated images are close to real images. CLIP score above 0.3 means images match text well. Images look sharp, colorful, and relevant.

Bad: FID above 100 means images look very different from real ones. CLIP score below 0.1 means images do not match the prompt. Images may be blurry, strange, or unrelated.

Common pitfalls in evaluating Stable Diffusion

• Overfitting: Model may memorize training images, producing less diverse outputs.
• Data leakage: Using test images in training can falsely improve metrics.
• Ignoring diversity: Only checking image quality but not variety can mislead about model performance.
• Misinterpreting metrics: Low FID alone does not guarantee good text-image match; use CLIP score too.

Self-check question

Your Stable Diffusion model has a FID of 25 but a CLIP score of 0.05. Is it good?

Answer: No, because while the images look realistic (low FID), they do not match the text prompts well (very low CLIP score). The model needs improvement to better understand and generate images that fit the text.

Key Result

Stable Diffusion quality is best judged by FID for image realism and CLIP score for text-image relevance.