Few-shot learning with prompts in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Few-shot learning with prompts

Which metric matters for Few-shot learning with prompts and WHY

In few-shot learning with prompts, the model sees very few examples before making predictions. Because of this, accuracy is important to see how often the model gets the right answer. But accuracy alone can be misleading if classes are unbalanced.

Therefore, precision and recall are also key. Precision tells us how many predicted answers are actually correct, and recall tells us how many correct answers the model finds out of all possible correct ones.

Since few-shot learning often deals with limited data, F1 score is very useful. It balances precision and recall into one number, showing overall quality.

For tasks like classification, a confusion matrix helps visualize where the model makes mistakes.

Confusion matrix example

Imagine a 3-class classification with few-shot prompts. Here is a confusion matrix from 100 samples:

      | Predicted A | Predicted B | Predicted C |
      |-------------|-------------|-------------|
      | True A: 30  | 5           | 0           |
      | True B: 3   | 25          | 2           |
      | True C: 1   | 4           | 30          |
    

Totals: TP (correct predictions) = 30 + 25 + 30 = 85, FP and FN are the off-diagonal numbers.

Precision vs Recall tradeoff with examples

In few-shot learning, sometimes the model guesses fewer answers to be sure (high precision, low recall). Other times it guesses more to catch all correct answers (high recall, low precision).

Example 1: A medical diagnosis prompt where missing a disease is dangerous. Here, high recall is more important to catch all cases, even if some false alarms happen.

Example 2: A spam detection prompt where marking good emails as spam is bad. Here, high precision is more important to avoid false positives.

Good vs Bad metric values for Few-shot learning with prompts

Good: Accuracy above 80%, precision and recall balanced above 75%, and F1 score close to both. Confusion matrix shows most predictions on the diagonal.

Bad: Accuracy near random chance (e.g., 33% for 3 classes), precision very low (many false positives), recall very low (many missed correct answers), and F1 score low. Confusion matrix shows many off-diagonal errors.

Common pitfalls in metrics for Few-shot learning with prompts

• Accuracy paradox: High accuracy can happen if one class dominates, hiding poor performance on others.
• Data leakage: If prompt examples leak answers, metrics look better but model is not truly learning.
• Overfitting: Model may memorize few-shot examples, showing high training metrics but poor real-world results.
• Small sample size: Few-shot means few examples, so metrics can vary a lot and be unstable.

Self-check question

Your few-shot prompt model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is this good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of positive cases, which is very risky. High accuracy is misleading because most data is negative. You need to improve recall to catch more positive cases.

Key Result

In few-shot learning with prompts, balanced precision, recall, and F1 score matter most to judge model quality beyond accuracy.

Practice

1. What is the main idea behind few-shot learning with prompts in NLP?

easy

A. Fine-tuning a model with many epochs on labeled data

B. Training a model with a large dataset from scratch

C. Using no examples and relying only on model pretraining

D. Using a few examples in a prompt to teach a model a new task

Solution

1. Step 1: Understand few-shot learning concept

   Few-shot learning means teaching a model with only a few examples instead of large datasets.

2. Step 2: Identify the role of prompts

   Prompts provide examples and instructions to guide the model on the new task.

3. Final Answer:

   Using a few examples in a prompt to teach a model a new task -> Option D

4. Quick Check:

   Few-shot learning = few examples + prompt [OK]

Hint: Few-shot means few examples in prompt, not full training [OK]

Common Mistakes:

• Confusing few-shot with full training
• Thinking no examples are used
• Mixing few-shot with fine-tuning

2. Which of the following is the correct way to format a few-shot prompt for sentiment analysis?

easy

A. "Sentiment: Positive Review: I love this movie!\nSentiment: Negative Review: This was bad.\nSentiment: Neutral Review: The plot was..."

B. "Review: I love this movie! Sentiment: Positive\nReview: This was bad. Sentiment: Negative\nReview: The plot was..."

C. "I love this movie! Positive\nThis was bad. Negative\nThe plot was... Neutral"

D. "Review: Positive I love this movie!\nReview: Negative This was bad.\nReview: Neutral The plot was..."

Solution

1. Step 1: Check prompt structure for clarity

   The prompt should clearly separate examples with labels and inputs, like 'Review: ... Sentiment: ...'.

2. Step 2: Identify correct example formatting

   "Review: I love this movie! Sentiment: Positive\nReview: This was bad. Sentiment: Negative\nReview: The plot was..." uses a consistent pattern: input first, then label, separated by new lines.

3. Final Answer:

   "Review: I love this movie! Sentiment: Positive\nReview: This was bad. Sentiment: Negative\nReview: The plot was..." -> Option B

4. Quick Check:

   Clear input-label pairs = "Review: I love this movie! Sentiment: Positive\nReview: This was bad. Sentiment: Negative\nReview: The plot was..." [OK]

Hint: Use clear input-label pairs separated by new lines [OK]

Common Mistakes:

• Mixing label before input
• Not separating examples clearly
• Omitting labels or inputs

3. Given this few-shot prompt for a model:

"Translate English to French:\nEnglish: Hello\nFrench: Bonjour\nEnglish: Thank you\nFrench: Merci\nEnglish: Good night\nFrench:"

What is the most likely model output?

medium

A. Bonne nuit

B. Bonsoir

C. Salut

D. Au revoir

Solution

1. Step 1: Understand prompt pattern

   The prompt shows English phrases with their French translations, ending with English: Good night and expecting French output.

2. Step 2: Recall correct French translation

   The French for "Good night" is "Bonne nuit", matching the pattern of direct translation.

3. Final Answer:

   Bonne nuit -> Option A

4. Quick Check:

   Good night = Bonne nuit [OK]

Hint: Match prompt pattern and translate phrase literally [OK]

Common Mistakes:

• Choosing a greeting instead of night farewell
• Confusing similar French words
• Ignoring prompt pattern

4. You wrote this few-shot prompt for a model to classify animals:

"Animal: Dog\nType: Mammal\nAnimal: Eagle\nType: Bird\nAnimal: Salmon\nType:"

The model outputs "Dog" instead of the expected animal type. What is the likely error?

medium

A. The prompt repeats the animal name instead of type

B. The model was not given any examples

C. The prompt lacks clear separation between examples

D. The model cannot handle few-shot prompts

Solution

1. Step 1: Analyze prompt structure

   The prompt shows pairs of Animal and Type, but the last line expects the type for Salmon.

2. Step 2: Identify output mismatch

   The model outputs "Dog" (an animal) instead of a type, likely because the prompt repeats animal names where types should be.

3. Final Answer:

   The prompt repeats the animal name instead of type -> Option A

4. Quick Check:

   Output animal instead of type = prompt error [OK]

Hint: Check if prompt asks for correct label, not repeated input [OK]

Common Mistakes:

• Assuming model can't handle few-shot
• Ignoring prompt formatting errors
• Thinking examples are missing

5. You want to build a few-shot prompt to classify movie reviews as Positive or Negative. Which approach will most likely improve model accuracy?

hard

A. Use only one example of a positive review to keep prompt short

B. Avoid giving any examples and rely on model knowledge

C. Include clear examples with both positive and negative reviews and their labels

D. Add unrelated text to make prompt longer

Solution

1. Step 1: Understand few-shot prompt impact

   Providing clear examples of both classes helps the model learn the task better.

2. Step 2: Evaluate options for accuracy

   Include clear examples with both positive and negative reviews and their labels gives balanced examples, improving accuracy; others reduce clarity or add noise.

3. Final Answer:

   Include clear examples with both positive and negative reviews and their labels -> Option C

4. Quick Check:

   Balanced examples = better accuracy [OK]

Hint: Use clear, balanced examples for best few-shot results [OK]

Common Mistakes:

• Using too few or unbalanced examples
• Relying on no examples
• Adding irrelevant information