Zero-shot prompting in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Zero-shot prompting

Which metric matters for Zero-shot prompting and WHY

Zero-shot prompting means asking a model to do a task it has never seen before. Because the model has no training examples for this task, we want to know how well it guesses correctly right away.

The key metric here is accuracy, which tells us the percentage of correct answers the model gives without any extra training. Accuracy is simple and clear for zero-shot tasks because we want to see if the model understands the task from just the prompt.

Sometimes, if the task is about finding specific items (like detecting spam), precision and recall also matter to understand if the model is careful or misses important cases.

Confusion matrix example for Zero-shot prompting

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Negative (FN) |
      | False Positive (FP) | True Negative (TN)  |

      Example:
      TP = 40, FP = 10, FN = 20, TN = 30
      Total samples = 100

      Accuracy = (TP + TN) / Total = (40 + 30) / 100 = 0.7 (70%)
      Precision = TP / (TP + FP) = 40 / (40 + 10) = 0.8 (80%)
      Recall = TP / (TP + FN) = 40 / (40 + 20) = 0.67 (67%)
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * 0.8 * 0.67 / (0.8 + 0.67) ≈ 0.73 (73%)
    

Precision vs Recall tradeoff in Zero-shot prompting

Imagine you ask the model to find spam emails without training it first (zero-shot). If the model marks many emails as spam, it might catch most spam (high recall) but also mark good emails as spam (low precision).

If the model is very careful and marks only very obvious spam, it will have high precision but might miss some spam emails (low recall).

Depending on what matters more, you choose to improve precision or recall. For zero-shot, understanding this tradeoff helps decide if the model guesses too broadly or too narrowly.

Good vs Bad metric values for Zero-shot prompting

Good: Accuracy above 70% means the model understands the task well without examples. Precision and recall above 70% show balanced and reliable predictions.

Bad: Accuracy below 50% means the model guesses worse than random. Very low precision (<50%) means many wrong positive guesses. Very low recall (<50%) means the model misses many true positives.

Common pitfalls in Zero-shot prompting metrics

• Accuracy paradox: If the data is mostly one class, high accuracy can be misleading (e.g., always guessing the majority class).
• Data leakage: If the model has seen similar tasks before, zero-shot results may be overestimated.
• Overfitting indicators: Not relevant here since zero-shot means no training, but repeated prompt tuning can cause overfitting.
• Ignoring class imbalance: If one class is rare, precision and recall give better insight than accuracy alone.

Self-check question

Your zero-shot model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is it good for production?

Answer: No. The model misses most positive cases (low recall), which is critical in fraud detection. High accuracy is misleading because most data is negative. You need to improve recall before using it in production.

Key Result

Accuracy shows overall correctness in zero-shot prompting, but precision and recall reveal if the model guesses carefully or misses key cases.

Practice

1. What is the main idea behind zero-shot prompting in AI?

easy

A. Training a model with many examples before testing

B. Fine-tuning a model with labeled data

C. Using a model only for image recognition tasks

D. Asking a model to perform a task using only instructions without examples

Solution

1. Step 1: Understand zero-shot prompting concept

   Zero-shot prompting means giving a model instructions to do a task without providing example inputs or outputs.

2. Step 2: Compare options to definition

   Only Asking a model to perform a task using only instructions without examples matches this idea. Options A, C, and D describe other AI methods.

3. Final Answer:

   Asking a model to perform a task using only instructions without examples -> Option D

4. Quick Check:

   Zero-shot prompting = instructions only [OK]

Hint: Zero-shot means no examples, just instructions [OK]

Common Mistakes:

• Confusing zero-shot with training on examples
• Thinking zero-shot needs fine-tuning
• Assuming zero-shot only works for images

2. Which of the following is the correct way to write a zero-shot prompt for a model to translate English to Spanish?

easy

A. "Translate the following sentence to Spanish: 'Hello, how are you?'"

B. "Here are examples: 'Hello' -> 'Hola', 'Goodbye' -> 'Adiós'. Translate 'Hello, how are you?'"

C. "Train the model with English-Spanish pairs before translating."

D. "Translate using a dictionary lookup for each word."

Solution

1. Step 1: Identify zero-shot prompt style

   Zero-shot prompts give instructions without examples or training data.

2. Step 2: Check options for instructions only

   "Translate the following sentence to Spanish: 'Hello, how are you?'" is a direct instruction without examples. "Here are examples: 'Hello' -> 'Hola', 'Goodbye' -> 'Adiós'. Translate 'Hello, how are you?'" includes examples, so it's not zero-shot. Options C and D describe other methods.

3. Final Answer:

   "Translate the following sentence to Spanish: 'Hello, how are you?'" -> Option A

4. Quick Check:

   Zero-shot prompt = instruction only [OK]

Hint: Zero-shot prompts have no examples, just clear instructions [OK]

Common Mistakes:

• Including examples in zero-shot prompts
• Confusing zero-shot with few-shot prompting
• Thinking training is needed for zero-shot

3. Given this zero-shot prompt to a language model:
"Summarize this text in one sentence: 'The cat sat on the mat because it was tired.'"
What is the most likely model output?

medium

A. "Because it was tired, the cat sat on the mat, and the dog barked."

B. "The cat sat on the mat."

C. "The cat was tired and sat on the mat."

D. ""

Solution

1. Step 1: Understand the prompt and task

   The prompt asks for a one-sentence summary of the given text.

2. Step 2: Evaluate options for correct summary

   "The cat was tired and sat on the mat." captures the main idea clearly and concisely. "The cat sat on the mat." is incomplete, missing the reason. "Because it was tired, the cat sat on the mat, and the dog barked." adds unrelated info. "" is empty, so invalid.

3. Final Answer:

   "The cat was tired and sat on the mat." -> Option C

4. Quick Check:

   Summary includes main points = "The cat was tired and sat on the mat." [OK]

Hint: Summaries keep main ideas, no extra details [OK]

Common Mistakes:

• Choosing incomplete or unrelated outputs
• Ignoring the instruction to summarize in one sentence
• Selecting empty or irrelevant answers

4. You wrote this zero-shot prompt:
"Explain the benefits of exercise"
But the model returns an error or unrelated text. What is the likely problem?

medium

A. The prompt is too vague or lacks clear instructions

B. The model requires example inputs and outputs

C. The prompt uses too many examples

D. The model cannot understand English

Solution

1. Step 1: Analyze the prompt clarity

   The prompt "Explain the benefits of exercise" is short but may be too vague or lacks detail for the model to respond well.

2. Step 2: Consider model requirements

   Zero-shot prompting works best with clear, simple instructions. The model does not require examples (so B is wrong). The prompt has no examples (so C is wrong). The model understanding English is assumed (A is unlikely).

3. Final Answer:

   The prompt is too vague or lacks clear instructions -> Option A

4. Quick Check:

   Clear instructions needed for zero-shot [OK]

Hint: Make prompts clear and specific to avoid errors [OK]

Common Mistakes:

• Assuming examples are always needed
• Ignoring prompt clarity
• Blaming model language understanding incorrectly

5. You want to use zero-shot prompting to classify customer reviews as positive or negative. Which prompt is best to get accurate results?

hard

A. "Train a model on labeled reviews before classifying."

B. "Classify this review as positive or negative: 'The product works great and arrived on time.'"

C. "Here are examples: 'Good' -> positive, 'Bad' -> negative. Classify: 'The product works great and arrived on time.'"

D. "Translate the review to another language before classifying."

Solution

1. Step 1: Identify zero-shot prompt requirements

   Zero-shot prompting uses instructions only, no examples or training.

2. Step 2: Evaluate prompt options

   "Classify this review as positive or negative: 'The product works great and arrived on time.'" is a clear instruction without examples, fitting zero-shot. "Here are examples: 'Good' -> positive, 'Bad' -> negative. Classify: 'The product works great and arrived on time.'" includes examples, so it's few-shot. "Train a model on labeled reviews before classifying." requires training, not zero-shot. "Translate the review to another language before classifying." is unrelated to classification.

3. Final Answer:

   "Classify this review as positive or negative: 'The product works great and arrived on time.'" -> Option B

4. Quick Check:

   Zero-shot = instruction only, no examples [OK]

Hint: Use clear instructions without examples for zero-shot tasks [OK]

Common Mistakes:

• Adding examples in zero-shot prompts
• Confusing zero-shot with training or few-shot
• Using unrelated steps like translation