Temperature and sampling parameters in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Temperature and sampling parameters

Which metric matters for Temperature and Sampling Parameters and WHY

When using temperature and sampling in generative AI, the key metrics are perplexity and diversity. Perplexity measures how well the model predicts the next word, showing if the output is sensible. Diversity measures how varied the generated text is, showing creativity. Temperature controls randomness: low temperature means safer, predictable text (lower perplexity, less diversity), high temperature means more creative but riskier text (higher perplexity, more diversity). Sampling parameters like top-k or nucleus sampling control how many options the model considers, affecting quality and variety. These metrics help balance making text both meaningful and interesting.

Confusion Matrix or Equivalent Visualization

For temperature and sampling, we don't use a confusion matrix like in classification. Instead, we look at probability distributions over possible next words.

    Example: Next word probabilities at different temperatures

    Temperature = 0.2 (low):
    word1: 0.7
    word2: 0.2
    word3: 0.1

    Temperature = 1.0 (medium):
    word1: 0.4
    word2: 0.35
    word3: 0.25

    Temperature = 2.0 (high):
    word1: 0.2
    word2: 0.3
    word3: 0.5

    As temperature rises, probabilities spread out, increasing randomness.
    

Precision vs Recall Tradeoff (or Equivalent) with Concrete Examples

Instead of precision and recall, temperature and sampling balance coherence vs creativity.

• Low temperature (e.g., 0.2): Output is very coherent and safe but can be repetitive or boring. Like always ordering the same meal at a restaurant.
• High temperature (e.g., 1.5): Output is creative and surprising but may be confusing or nonsensical. Like trying a new exotic dish that might not taste good.

Sampling parameters like top-k limit choices to top words, reducing weird outputs but also limiting creativity.

What "Good" vs "Bad" Metric Values Look Like for This Use Case

Good: Perplexity is moderate, showing the model predicts well but still allows some surprise. Diversity is balanced, so text is interesting but understandable. For example, temperature around 0.7 to 1.0 often works well.

Bad: Very low temperature (near 0) leads to dull, repetitive text (low diversity). Very high temperature (above 1.5) causes gibberish or off-topic text (high perplexity, too much diversity). Sampling with too small top-k can make output too narrow; too large can make it noisy.

Metrics Pitfalls

• Ignoring context: High diversity is not always good if it breaks meaning.
• Overfitting to training data: Low temperature might hide that model just repeats training phrases.
• Misinterpreting perplexity: Lower perplexity means better prediction but not always better creativity.
• Sampling bias: Using fixed top-k without tuning can limit output quality.

Self Check

Your generative model uses temperature 1.5 and top-k 50. The output is very creative but often off-topic and confusing. Is this good for production? Why or why not?

Answer: No, it is not good for production because the high temperature and large top-k cause too much randomness, making the output confusing and less useful. You should lower temperature or top-k to improve coherence.

Key Result

Temperature and sampling balance prediction quality (perplexity) and creativity (diversity) to produce meaningful yet interesting text.

Practice

1. What does the temperature parameter control in AI text generation?

easy

A. The speed of the AI's response

B. The length of the generated text

C. How random or focused the AI's answers are

D. The number of words the AI can use

Solution

1. Step 1: Understand the role of temperature

   The temperature parameter adjusts randomness in AI output. A low temperature makes answers more focused and predictable, while a high temperature increases randomness and creativity.

2. Step 2: Match the description to the options

   Only How random or focused the AI's answers are correctly describes temperature as controlling randomness or focus in AI answers.

3. Final Answer:

   How random or focused the AI's answers are -> Option C

4. Quick Check:

   Temperature controls randomness = A [OK]

Hint: Temperature means randomness level in AI output [OK]

Common Mistakes:

• Confusing temperature with output length
• Thinking temperature controls speed
• Mixing temperature with vocabulary size

2. Which of the following is the correct way to set the temperature to 0.7 in a Python API call for text generation?

easy

A. generate_text(temperature=0.7)

B. generate_text(temp=7)

C. generate_text(temperature='0.7')

D. generate_text(temperature=7)

Solution

1. Step 1: Identify correct parameter name and type

   The parameter controlling randomness is named temperature and expects a float value between 0 and 1 (commonly).

2. Step 2: Check each option

   generate_text(temperature=0.7) uses the correct parameter name and a valid float value 0.7. generate_text(temp=7) uses wrong parameter name and integer 7. generate_text(temperature='0.7') passes a string instead of float. generate_text(temperature=7) uses an invalid integer 7 instead of a float between 0 and 1.

3. Final Answer:

   generate_text(temperature=0.7) -> Option A

4. Quick Check:

   Correct parameter and float value = D [OK]

Hint: Use parameter name 'temperature' with float value [OK]

Common Mistakes:

• Using wrong parameter name like 'temp'
• Passing temperature as string instead of float
• Using values outside 0-1 range

3. Given this code snippet:

response = generate_text(prompt='Hello', temperature=0.1, top_p=0.9)
print(response)

What is the expected behavior of the AI output?

medium

A. Highly random and creative text

B. Very focused and predictable text

C. Text with many rare words

D. Text ignoring the prompt

Solution

1. Step 1: Analyze temperature value

   A temperature of 0.1 is very low, so the AI output will be focused and predictable, avoiding randomness.

2. Step 2: Analyze top_p value

   Top-p of 0.9 means the AI considers the most probable words covering 90% probability, further limiting randomness.

3. Final Answer:

   Very focused and predictable text -> Option B

4. Quick Check:

   Low temperature + top_p = focused output [OK]

Hint: Low temperature means focused output, not random [OK]

Common Mistakes:

• Assuming low temperature means more creativity
• Ignoring top_p effect on word choice
• Thinking AI ignores prompt with low temperature

4. You set temperature=1.5 in your AI call but get an error. What is the likely cause and fix?

medium

A. Temperature must be an integer; set it to 1

B. Temperature must be a string; set it to '1.5'

C. Temperature cannot be set; remove it

D. Temperature must be between 0 and 1; set it to 0.9

Solution

1. Step 1: Understand valid temperature range

   Temperature values are usually required to be between 0 and 1 to control randomness properly.

2. Step 2: Identify error cause and fix

   Setting temperature to 1.5 is outside the valid range, causing an error. Fix is to set it to a valid value like 0.9.

3. Final Answer:

   Temperature must be between 0 and 1; set it to 0.9 -> Option D

4. Quick Check:

   Temperature range 0-1 = B [OK]

Hint: Temperature must be 0 to 1, not above [OK]

Common Mistakes:

• Using values above 1 for temperature
• Thinking temperature must be integer
• Passing temperature as string

5. You want the AI to generate creative but coherent stories. Which combination of temperature and top_p is best?

hard

A. temperature=0.8, top_p=0.9

B. temperature=0.2, top_p=0.5

C. temperature=1.0, top_p=0.1

D. temperature=0.0, top_p=1.0

Solution

1. Step 1: Understand desired output style

   Creative but coherent stories need some randomness (creativity) but also focus (coherence).

2. Step 2: Evaluate each parameter combination

   temperature=0.8, top_p=0.9 has temperature 0.8 (moderately high randomness) and top_p 0.9 (limits to likely words), balancing creativity and coherence. temperature=0.2, top_p=0.5 is too low randomness, temperature=1.0, top_p=0.1 has very low top_p causing incoherence, temperature=0.0, top_p=1.0 is zero randomness, so very dull.

3. Final Answer:

   temperature=0.8, top_p=0.9 -> Option A

4. Quick Check:

   Moderate temperature + high top_p = creative & coherent [OK]

Hint: Use moderate temperature and high top_p for creative coherence [OK]

Common Mistakes:

• Choosing zero temperature for creativity
• Using very low top_p causing odd word choices
• Setting temperature too high causing nonsense