What Generative AI actually is in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - What Generative AI actually is

Which metric matters for this concept and WHY

For Generative AI, quality of output is key. Metrics like Perplexity measure how well the model predicts text, showing if it understands language patterns. BLEU or ROUGE scores compare generated text to human examples, checking if output is meaningful and relevant. For images, FID score measures how close generated images are to real ones. These metrics matter because they tell us if the AI creates believable and useful content.

Confusion matrix or equivalent visualization (ASCII)

Generative AI does not use a confusion matrix like classifiers. Instead, we look at example outputs and scores:

    Example: Text generation quality scores
    ---------------------------------------
    Model output: "The cat sat on the mat."
    Reference: "The cat is sitting on the mat."

    BLEU score: 0.75 (higher is better)
    Perplexity: 12.3 (lower is better)
    

For image generation, FID score example:

    FID score: 25.4 (lower means generated images look more like real ones)
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

Generative AI tradeoffs are different from classification. Here, we balance creativity and accuracy.

• High creativity, low accuracy: AI makes new ideas but may produce nonsense or errors.
• High accuracy, low creativity: AI repeats known patterns but output is safe and reliable.

Example: A story generator that invents new plots (creative) vs one that copies training stories exactly (accurate but boring).

What "good" vs "bad" metric values look like for this use case

Good generative AI metrics mean:

• Low perplexity: Model predicts text well, so output is fluent.
• High BLEU/ROUGE: Output matches human examples closely.
• Low FID: Generated images look realistic.

Bad metrics mean:

• High perplexity: Output is confusing or unnatural.
• Low BLEU/ROUGE: Output is irrelevant or off-topic.
• High FID: Images look fake or distorted.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Overfitting: Model memorizes training data, producing perfect but copied outputs, not creative ones.
• Data leakage: If test data is in training, metrics look better but model is not truly generative.
• Metric mismatch: BLEU or ROUGE may not capture creativity or meaning well.
• Perplexity limits: Low perplexity doesn't guarantee interesting or useful output.

Self-check: Your model has low perplexity but low BLEU score. Is it good?

No, this means the model predicts text well (low perplexity) but its output does not match human examples closely (low BLEU). It might produce fluent but irrelevant or generic text. So, it is not good for tasks needing meaningful or accurate content.

Key Result

Generative AI quality is best judged by metrics like perplexity, BLEU, and FID that measure fluency, relevance, and realism.

Practice

1. What is the main purpose of Generative AI?

easy

A. To store large amounts of data efficiently

B. To delete irrelevant information from datasets

C. To only classify existing data into categories

D. To create new content by learning from examples

Solution

1. Step 1: Understand the role of Generative AI

   Generative AI learns patterns from data and creates new content based on those patterns.

2. Step 2: Compare options with the definition

   Only To create new content by learning from examples describes creating new content by learning from examples, which matches the main purpose.

3. Final Answer:

   To create new content by learning from examples -> Option D

4. Quick Check:

   Generative AI = create new content [OK]

Hint: Generative AI makes new stuff from learned data [OK]

Common Mistakes:

• Confusing Generative AI with data storage
• Thinking it only classifies data
• Believing it deletes data

2. Which of the following is the correct way to describe Generative AI in simple code terms?

easy

A. Train a model, then generate new outputs

B. Only collect data without processing

C. Manually write all new content

D. Delete old models before training

Solution

1. Step 1: Identify the typical workflow of Generative AI

   Generative AI involves training a model on data and then using it to create new outputs.

2. Step 2: Match options to this workflow

   Train a model, then generate new outputs correctly states this process, while others describe unrelated or incorrect actions.

3. Final Answer:

   Train a model, then generate new outputs -> Option A

4. Quick Check:

   Train then generate = correct process [OK]

Hint: Generative AI = train model + create new data [OK]

Common Mistakes:

• Thinking Generative AI only collects data
• Assuming manual content creation is AI
• Confusing training with deleting models

3. Consider this Python-like pseudocode for a simple Generative AI process:

model = train(data)
new_content = model.generate()

What will new_content most likely contain?

medium

A. A new example similar to the training data

B. The original training data unchanged

C. An error message because generate() is undefined

D. An empty output with no content

Solution

1. Step 1: Understand the code steps

   The code trains a model on data, then calls generate() to create new content.

2. Step 2: Predict the output of generate()

   Generate() produces new content similar to what the model learned, not the original data or errors.

3. Final Answer:

   A new example similar to the training data -> Option A

4. Quick Check:

   generate() = new similar content [OK]

Hint: generate() creates new data like training examples [OK]

Common Mistakes:

• Thinking generate() returns original data
• Assuming generate() causes an error
• Expecting empty output

4. The following code is intended to train a Generative AI model and generate new content:

model = train(data)
new_content = model.generate(data)

What is the likely problem here?

medium

A. model should be a list, not a model object

B. train() should not take data as input

C. generate() should not take data as input after training

D. new_content should be assigned before training

Solution

1. Step 1: Review typical usage of generate()

   After training, generate() usually creates new content without needing input data again.

2. Step 2: Identify misuse in code

   Passing data to generate() is incorrect; it should generate based on learned patterns alone.

3. Final Answer:

   generate() should not take data as input after training -> Option C

4. Quick Check:

   generate() no input needed [OK]

Hint: generate() uses learned model, no extra data input [OK]

Common Mistakes:

• Thinking train() shouldn't take data
• Confusing model type
• Assigning new_content before training

5. You want to create a Generative AI that writes short poems. Which steps best describe the process?

hard

A. Write poems manually, then use AI to classify them

B. Collect poem examples, train model on them, generate new poems

C. Train model on random text, then delete training data

D. Generate poems first, then collect examples to train

Solution

1. Step 1: Understand the goal of Generative AI for poems

   The AI needs to learn from existing poems to create new ones.

2. Step 2: Identify the correct sequence of actions

   Collecting examples, training the model, then generating new poems is the correct order.

3. Final Answer:

   Collect poem examples, train model on them, generate new poems -> Option B

4. Quick Check:

   Learn from examples, then create new [OK]

Hint: Train on examples first, then generate new content [OK]

Common Mistakes:

• Trying to generate before training
• Confusing classification with generation
• Deleting training data too early