Why Generative AI is transforming technology in Prompt Engineering / GenAI - Why Metrics Matter

Metrics & Evaluation - Why Generative AI is transforming technology

Which metric matters for this concept and WHY

For generative AI, key metrics include perplexity and BLEU score for language models, and FID (Fréchet Inception Distance) for image generation. These metrics measure how well the AI creates realistic and meaningful outputs. Perplexity shows how well the model predicts text, BLEU compares generated text to human examples, and FID measures image quality and diversity. These metrics matter because they tell us if the AI is producing useful and believable content, which is the core of generative AI's impact.

Confusion matrix or equivalent visualization (ASCII)

Generative AI does not use a traditional confusion matrix because it creates new data rather than classifying existing data. Instead, evaluation uses metrics like BLEU or FID scores. Here is an example of a BLEU score comparison:

Reference: "The cat sits on the mat."
Generated: "The cat is sitting on the mat."
BLEU score: 0.85 (high similarity)

Reference: "The cat sits on the mat."
Generated: "A dog runs outside."
BLEU score: 0.10 (low similarity)
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

In generative AI, the tradeoff is often between creativity and accuracy. For example, a text generator can produce very accurate sentences (high accuracy) but may be boring or repetitive (low creativity). Or it can create very novel and diverse sentences (high creativity) but sometimes make mistakes or produce irrelevant content (low accuracy). Balancing these helps make generative AI useful and engaging.

Example: A chatbot that only repeats facts (high accuracy) might feel dull, while one that invents stories (high creativity) might sometimes say wrong things. The best models find a good middle ground.

What "good" vs "bad" metric values look like for this use case

Good generative AI metrics mean:

• Low perplexity (better text prediction)
• High BLEU score (close to human text)
• Low FID score (high-quality, realistic images)

Bad metrics mean:

• High perplexity (confused text generation)
• Low BLEU score (text far from human examples)
• High FID score (blurry or unrealistic images)

Good metrics show the AI is learning patterns well and creating believable content. Bad metrics show the AI is struggling or producing poor results.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

Common pitfalls in generative AI metrics include:

• Overfitting: The model memorizes training data and repeats it instead of creating new content. This can look like very good scores but poor creativity.
• Data leakage: If test data is too similar to training data, metrics may be falsely high.
• Accuracy paradox: A model might score well on simple metrics but produce nonsensical or irrelevant content.
• Ignoring diversity: Metrics may not capture if the AI generates varied outputs, leading to dull or repetitive results.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

This question is about fraud detection, not generative AI, but it teaches an important lesson. A model with 98% accuracy but only 12% recall on fraud means it misses most fraud cases. This is bad because catching fraud (high recall) is critical. Similarly, in generative AI, a model might score well on some metrics but fail in important ways like creativity or relevance. Always check multiple metrics to understand true performance.

Key Result

Generative AI success depends on balanced metrics like low perplexity, high BLEU, and low FID to ensure realistic and creative outputs.

Practice

1. What is the main reason Generative AI is transforming technology?

easy

A. It only stores large amounts of data efficiently.

B. It can create new content automatically, saving time and effort.

C. It replaces all human jobs immediately.

D. It only works with numbers and calculations.

Solution

1. Step 1: Understand the core function of Generative AI

   Generative AI is designed to create new content automatically, such as text, images, or music.

2. Step 2: Compare options with this function

   Only It can create new content automatically, saving time and effort. describes this key feature, while others describe unrelated or incorrect ideas.

3. Final Answer:

   It can create new content automatically, saving time and effort. -> Option B

4. Quick Check:

   Generative AI creates content = A [OK]

Hint: Focus on content creation as the key feature [OK]

Common Mistakes:

• Thinking it only stores data
• Believing it replaces all jobs instantly
• Assuming it only does calculations

2. Which of the following is the correct way to describe Generative AI's role?

easy

A. Generative AI only analyzes existing data without creating anything new.

B. Generative AI is used only for simple math calculations.

C. Generative AI helps create new content like writing and art automatically.

D. Generative AI is a tool for storing large databases.

Solution

1. Step 1: Identify the role of Generative AI

   Generative AI is known for creating new content such as text, images, and designs automatically.

2. Step 2: Match the correct description

   Generative AI helps create new content like writing and art automatically. correctly states this role, while others describe unrelated or incorrect functions.

3. Final Answer:

   Generative AI helps create new content like writing and art automatically. -> Option C

4. Quick Check:

   Role = content creation = D [OK]

Hint: Look for 'creating new content' in the option [OK]

Common Mistakes:

• Confusing analysis with creation
• Thinking it only stores data
• Assuming it does only calculations

3. Consider this Python code simulating a simple Generative AI output process:

def generate_text(seed):
    return seed + ' world!'

output = generate_text('Hello')
print(output)

What will be printed?

medium

A. Hello

B. Hello world

C. world!

D. Hello world!

Solution

1. Step 1: Understand the function generate_text

   The function adds the string ' world!' to the input seed string.

2. Step 2: Apply the function to 'Hello'

   Calling generate_text('Hello') returns 'Hello world!'.

3. Final Answer:

   Hello world! -> Option D

4. Quick Check:

   Concatenate 'Hello' + ' world!' = 'Hello world!' [OK]

Hint: Check string concatenation carefully [OK]

Common Mistakes:

• Ignoring the added ' world!'
• Confusing output with input
• Missing the exclamation mark

4. This code tries to generate a list of AI-generated texts but has an error:

texts = []
for i in range(3):
    texts.append('Text ' + i)
print(texts)

What is the error and how to fix it?

medium

A. TypeError because 'i' is int; fix by converting i to string with str(i).

B. SyntaxError due to missing colon after for loop.

C. NameError because 'texts' is not defined.

D. No error; code runs fine.

Solution

1. Step 1: Identify the error in string concatenation

   The code tries to add a string and an integer, which causes a TypeError.

2. Step 2: Fix by converting integer to string

   Use str(i) to convert the integer i to a string before concatenation.

3. Final Answer:

   TypeError because 'i' is int; fix by converting i to string with str(i). -> Option A

4. Quick Check:

   String + int causes error; convert int to string [OK]

Hint: Remember to convert numbers to strings before concatenation [OK]

Common Mistakes:

• Ignoring type mismatch
• Thinking syntax is wrong
• Assuming variable is undefined

5. You want to use Generative AI to help design a logo automatically. Which approach best uses this technology?

hard

A. Train a model to generate new logo images based on examples you provide.

B. Manually draw logos and save them as files.

C. Use a calculator to measure logo dimensions.

D. Store existing logos in a database without changes.

Solution

1. Step 1: Understand Generative AI's application in design

   Generative AI can create new images by learning from example logos.

2. Step 2: Identify the option that uses AI generation

   Train a model to generate new logo images based on examples you provide. describes training a model to generate new logos, which fits the use of Generative AI.

3. Final Answer:

   Train a model to generate new logo images based on examples you provide. -> Option A

4. Quick Check:

   Use AI to generate new designs = A [OK]

Hint: Choose option involving AI creating new content [OK]

Common Mistakes:

• Confusing manual work with AI generation
• Thinking storing data is generation
• Using unrelated tools like calculators