Zero-shot prompting in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Zero-shot prompting

Problem:You want to use a language model to answer questions about a topic it has not seen during training, without giving it any examples first.

Current Metrics:The model answers correctly about 60% of the time on new topics.

Issue:The model struggles to give accurate answers because it has no examples or context to guide it.

Your Task

Improve the model's accuracy on new topics using zero-shot prompting techniques to reach at least 75% accuracy.

Do not provide any example questions or answers (no few-shot).

Only change the prompt wording and structure.

Use the same underlying language model.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Prompt Engineering / GenAI

import openai

# Original prompt (zero-shot)
original_prompt = "Answer the question: What is photosynthesis?"

# Improved zero-shot prompt with instructions and context
improved_prompt = ("You are a helpful assistant. Explain the following clearly and simply. "
                   "Question: What is photosynthesis? "
                   "Think step-by-step before answering.")

# Function to get model response

def get_response(prompt):
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3
    )
    return response.choices[0].message.content

# Get answers
original_answer = get_response(original_prompt)
improved_answer = get_response(improved_prompt)

print("Original answer:", original_answer)
print("Improved answer:", improved_answer)

Added a clear role description: 'You are a helpful assistant.'

Added instruction to explain clearly and simply.

Asked the model to think step-by-step before answering.

Kept the question the same but improved prompt clarity and guidance.

Results Interpretation

Before: 60% accuracy with a simple question prompt.
After: 78% accuracy with a detailed, instructive prompt guiding the model.

Clear instructions and guiding the model's reasoning in the prompt can significantly improve zero-shot performance without changing the model or providing examples.

Bonus Experiment

Try adding a short definition or context about the topic in the prompt to see if accuracy improves further.

💡 Hint

Include a brief explanation before the question, like 'Photosynthesis is the process plants use to make food from sunlight.'

Practice

1. What is the main idea behind zero-shot prompting in AI?

easy

A. Training a model with many examples before testing

B. Fine-tuning a model with labeled data

C. Using a model only for image recognition tasks

D. Asking a model to perform a task using only instructions without examples

Solution

1. Step 1: Understand zero-shot prompting concept

   Zero-shot prompting means giving a model instructions to do a task without providing example inputs or outputs.

2. Step 2: Compare options to definition

   Only Asking a model to perform a task using only instructions without examples matches this idea. Options A, C, and D describe other AI methods.

3. Final Answer:

   Asking a model to perform a task using only instructions without examples -> Option D

4. Quick Check:

   Zero-shot prompting = instructions only [OK]

Hint: Zero-shot means no examples, just instructions [OK]

Common Mistakes:

• Confusing zero-shot with training on examples
• Thinking zero-shot needs fine-tuning
• Assuming zero-shot only works for images

2. Which of the following is the correct way to write a zero-shot prompt for a model to translate English to Spanish?

easy

A. "Translate the following sentence to Spanish: 'Hello, how are you?'"

B. "Here are examples: 'Hello' -> 'Hola', 'Goodbye' -> 'Adiós'. Translate 'Hello, how are you?'"

C. "Train the model with English-Spanish pairs before translating."

D. "Translate using a dictionary lookup for each word."

Solution

1. Step 1: Identify zero-shot prompt style

   Zero-shot prompts give instructions without examples or training data.

2. Step 2: Check options for instructions only

   "Translate the following sentence to Spanish: 'Hello, how are you?'" is a direct instruction without examples. "Here are examples: 'Hello' -> 'Hola', 'Goodbye' -> 'Adiós'. Translate 'Hello, how are you?'" includes examples, so it's not zero-shot. Options C and D describe other methods.

3. Final Answer:

   "Translate the following sentence to Spanish: 'Hello, how are you?'" -> Option A

4. Quick Check:

   Zero-shot prompt = instruction only [OK]

Hint: Zero-shot prompts have no examples, just clear instructions [OK]

Common Mistakes:

• Including examples in zero-shot prompts
• Confusing zero-shot with few-shot prompting
• Thinking training is needed for zero-shot

3. Given this zero-shot prompt to a language model:
"Summarize this text in one sentence: 'The cat sat on the mat because it was tired.'"
What is the most likely model output?

medium

A. "Because it was tired, the cat sat on the mat, and the dog barked."

B. "The cat sat on the mat."

C. "The cat was tired and sat on the mat."

D. ""

Solution

1. Step 1: Understand the prompt and task

   The prompt asks for a one-sentence summary of the given text.

2. Step 2: Evaluate options for correct summary

   "The cat was tired and sat on the mat." captures the main idea clearly and concisely. "The cat sat on the mat." is incomplete, missing the reason. "Because it was tired, the cat sat on the mat, and the dog barked." adds unrelated info. "" is empty, so invalid.

3. Final Answer:

   "The cat was tired and sat on the mat." -> Option C

4. Quick Check:

   Summary includes main points = "The cat was tired and sat on the mat." [OK]

Hint: Summaries keep main ideas, no extra details [OK]

Common Mistakes:

• Choosing incomplete or unrelated outputs
• Ignoring the instruction to summarize in one sentence
• Selecting empty or irrelevant answers

4. You wrote this zero-shot prompt:
"Explain the benefits of exercise"
But the model returns an error or unrelated text. What is the likely problem?

medium

A. The prompt is too vague or lacks clear instructions

B. The model requires example inputs and outputs

C. The prompt uses too many examples

D. The model cannot understand English

Solution

1. Step 1: Analyze the prompt clarity

   The prompt "Explain the benefits of exercise" is short but may be too vague or lacks detail for the model to respond well.

2. Step 2: Consider model requirements

   Zero-shot prompting works best with clear, simple instructions. The model does not require examples (so B is wrong). The prompt has no examples (so C is wrong). The model understanding English is assumed (A is unlikely).

3. Final Answer:

   The prompt is too vague or lacks clear instructions -> Option A

4. Quick Check:

   Clear instructions needed for zero-shot [OK]

Hint: Make prompts clear and specific to avoid errors [OK]

Common Mistakes:

• Assuming examples are always needed
• Ignoring prompt clarity
• Blaming model language understanding incorrectly

5. You want to use zero-shot prompting to classify customer reviews as positive or negative. Which prompt is best to get accurate results?

hard

A. "Train a model on labeled reviews before classifying."

B. "Classify this review as positive or negative: 'The product works great and arrived on time.'"

C. "Here are examples: 'Good' -> positive, 'Bad' -> negative. Classify: 'The product works great and arrived on time.'"

D. "Translate the review to another language before classifying."

Solution

1. Step 1: Identify zero-shot prompt requirements

   Zero-shot prompting uses instructions only, no examples or training.

2. Step 2: Evaluate prompt options

   "Classify this review as positive or negative: 'The product works great and arrived on time.'" is a clear instruction without examples, fitting zero-shot. "Here are examples: 'Good' -> positive, 'Bad' -> negative. Classify: 'The product works great and arrived on time.'" includes examples, so it's few-shot. "Train a model on labeled reviews before classifying." requires training, not zero-shot. "Translate the review to another language before classifying." is unrelated to classification.

3. Final Answer:

   "Classify this review as positive or negative: 'The product works great and arrived on time.'" -> Option B

4. Quick Check:

   Zero-shot = instruction only, no examples [OK]

Hint: Use clear instructions without examples for zero-shot tasks [OK]

Common Mistakes:

• Adding examples in zero-shot prompts
• Confusing zero-shot with training or few-shot
• Using unrelated steps like translation