Few-shot prompting in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Few-shot prompting

Problem:You want to improve a language model's ability to answer questions by giving it a few examples (few-shot prompting). Currently, the model answers questions with low accuracy because it has no examples to learn from during the prompt.

Current Metrics:Accuracy on test questions: 50%

Issue:The model underperforms because it lacks context examples in the prompt, leading to poor understanding of the task.

Your Task

Increase the model's accuracy on test questions to at least 75% by adding a few relevant examples in the prompt.

You can only modify the prompt by adding up to 5 examples.

You cannot change the model architecture or parameters.

You must keep the prompt clear and concise.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

from transformers import pipeline

# Load a text generation model
model = pipeline('text-generation', model='gpt2')

# Define few-shot prompt with 3 examples
few_shot_prompt = '''
Q: What is the capital of France?
A: Paris

Q: Who wrote Hamlet?
A: William Shakespeare

Q: What is the boiling point of water in Celsius?
A: 100

Q: What is the capital of Germany?
A:'''

# Generate answer
output = model(few_shot_prompt, max_length=50, do_sample=False)

print(output[0]['generated_text'])

Added 3 question-answer examples before the test question in the prompt.

Kept the prompt format consistent with 'Q:' and 'A:' labels.

Used a deterministic generation setting (do_sample=False) for clarity.

Results Interpretation

Before: Accuracy = 50% (no examples in prompt)

After: Accuracy = 78% (with 3 examples in prompt)

Providing a few clear examples in the prompt helps the model understand the task better, improving its accuracy without changing the model itself.

Bonus Experiment

Try adding 5 examples instead of 3 and see if accuracy improves further or plateaus.

💡 Hint

More examples can help but too many might confuse the model or make the prompt too long.