Instruction formatting in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Instruction formatting

Which metric matters for Instruction formatting and WHY

When working with instruction formatting in AI models, the key metric to focus on is accuracy. This measures how well the model follows the instructions given. If the model misunderstands or misformats the instructions, the output will be wrong or confusing.

Accuracy matters because the goal is to get the model to produce exactly what the instruction asks for. Other metrics like precision or recall are less relevant here because we want the entire instruction to be correctly followed, not just parts of it.

Confusion matrix or equivalent visualization

Instruction Followed Correctly | Instruction Followed Incorrectly
------------------------------|------------------------------
True Positive (TP): Correctly formatted instructions  | False Negative (FN): Instructions not followed
False Positive (FP): Incorrect formatting accepted as correct | True Negative (TN): Not applicable here

Total instructions = TP + FP + FN
    

In instruction formatting, TP means the model output matches the instruction perfectly. FN means the model failed to follow the instruction. FP and TN are less common but can represent cases where incorrect formatting is mistakenly accepted.

Precision vs Recall tradeoff with examples

In instruction formatting, precision means how often the model's formatted output is actually correct when it claims to be correct.

Recall means how many of the instructions the model correctly formats out of all instructions given.

For example, if a model formats 90 outputs and 80 are correct (precision ~89%), but it only correctly formats 80 out of 100 instructions (recall 80%), it means it is careful but misses some instructions.

Depending on the use case, you might want higher recall (follow all instructions even if some are imperfect) or higher precision (only produce output when very sure it is correct).

What "good" vs "bad" metric values look like for instruction formatting

• Good: Accuracy above 95%, precision and recall both high (above 90%). This means the model follows instructions well and rarely makes mistakes.
• Bad: Accuracy below 70%, precision or recall very low (below 50%). This means the model often misunderstands or misformats instructions.
• Balanced precision and recall are important. High precision but low recall means many instructions are ignored. High recall but low precision means many outputs are wrong.

Common pitfalls in instruction formatting metrics

• Accuracy paradox: If instructions are very simple or repetitive, a model might get high accuracy by guessing common patterns but fail on new instructions.
• Data leakage: If the model sees test instructions during training, metrics will be unrealistically high.
• Overfitting: The model might memorize specific instructions but fail to generalize to new ones, causing poor real-world performance.
• Ignoring partial correctness: Sometimes outputs partially follow instructions. Metrics that only count perfect matches miss this nuance.

Self-check question

Your model has 98% accuracy but only 12% recall on following instructions. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy likely means the model is correct when it tries, but the very low recall means it only follows a small fraction of instructions. This means many instructions are ignored, which is a big problem for instruction formatting.

Key Result

Accuracy is key for instruction formatting; balanced precision and recall ensure instructions are followed correctly and consistently.

Practice

1. What is the main purpose of instruction formatting when interacting with an AI?

easy

A. To help the AI understand your request clearly

B. To make the AI run faster

C. To change the AI's programming language

D. To limit the AI's response length

Solution

1. Step 1: Understand the role of instruction formatting

   Instruction formatting is about how you write your request to the AI so it understands what you want.

2. Step 2: Identify the correct purpose

   The main goal is to make the AI understand your request clearly, not to affect speed or language.

3. Final Answer:

   To help the AI understand your request clearly -> Option A

4. Quick Check:

   Instruction formatting = clear AI understanding [OK]

Hint: Think: Why do I write instructions clearly? To be understood [OK]

Common Mistakes:

• Confusing formatting with AI speed
• Thinking it changes AI's programming
• Assuming it limits response length

2. Which of the following is the correct way to format an instruction for an AI to write a short story about a dog?

easy

A. Story dog short please write quickly

B. dog story short write

C. Write a short story about a dog.

D. Write dog story short and fast

Solution

1. Step 1: Check clarity and grammar

   Write a short story about a dog. is a clear, simple sentence that tells the AI exactly what to do.

2. Step 2: Compare other options

   Options B, C, and D are unclear or awkward, making it hard for AI to understand.

3. Final Answer:

   Write a short story about a dog. -> Option C

4. Quick Check:

   Clear, simple sentence = Write a short story about a dog. [OK]

Hint: Choose the option with clear, complete sentences [OK]

Common Mistakes:

• Using incomplete or jumbled words
• Adding unnecessary words that confuse AI
• Ignoring sentence structure

3. Given the instruction: Summarize the following text in 2 sentences. What will the AI most likely produce?

medium

A. A detailed essay about the text

B. A summary in exactly two sentences

C. A list of keywords from the text

D. The original text repeated twice

Solution

1. Step 1: Understand the instruction

   The instruction asks for a summary limited to 2 sentences.

2. Step 2: Predict AI output

   The AI will produce a concise summary exactly in 2 sentences, not an essay or keywords.

3. Final Answer:

   A summary in exactly two sentences -> Option B

4. Quick Check:

   Instruction = summary in 2 sentences [OK]

Hint: Look for keywords like 'summarize' and '2 sentences' [OK]

Common Mistakes:

• Confusing summary with essay
• Expecting keywords instead of sentences
• Thinking AI repeats text

4. You gave the AI this instruction: Explain machine learning simple. The AI gives a confusing answer. What is the best fix?

medium

A. Make the instruction shorter.

B. Add more technical terms to the instruction.

C. Ask the AI to explain in a different language.

D. Change instruction to 'Explain machine learning in simple words.'

Solution

1. Step 1: Identify the problem with the instruction

   The original instruction is unclear and grammatically incomplete.

2. Step 2: Choose the clearer instruction

   Change instruction to 'Explain machine learning in simple words.' improves clarity by adding 'in simple words', making it easier for AI to understand.

3. Final Answer:

   Change instruction to 'Explain machine learning in simple words.' -> Option D

4. Quick Check:

   Clear, complete instruction = Change instruction to 'Explain machine learning in simple words.' [OK]

Hint: Make instructions clear and grammatically correct [OK]

Common Mistakes:

• Adding confusing technical terms
• Shortening instructions too much
• Changing language without reason

5. You want the AI to generate a list of 5 healthy breakfast ideas with calories under 300 each. Which instruction is best formatted?

hard

A. List 5 healthy breakfast ideas with calories under 300 each, including ingredients.

B. Give me breakfast ideas.

C. List 5 breakfast foods.

D. Tell me about healthy food.

Solution

1. Step 1: Identify key details needed

   The instruction must specify number, healthiness, calorie limit, and ingredients.

2. Step 2: Compare options for completeness

   List 5 healthy breakfast ideas with calories under 300 each, including ingredients. includes all details clearly; others are too vague or incomplete.

3. Final Answer:

   List 5 healthy breakfast ideas with calories under 300 each, including ingredients. -> Option A

4. Quick Check:

   Complete, detailed instruction = List 5 healthy breakfast ideas with calories under 300 each, including ingredients. [OK]

Hint: Include all details clearly in your instruction [OK]

Common Mistakes:

• Giving vague or incomplete instructions
• Missing calorie or quantity details
• Not specifying required output format