Temperature and sampling in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Temperature and sampling

Which metric matters for Temperature and Sampling and WHY

In language generation, we want to measure how well the model creates text that is both coherent and diverse. Metrics like perplexity show how well the model predicts the next word, but they don't capture creativity or variety.

Instead, we look at diversity metrics such as distinct-n (how many unique n-grams appear) and human evaluation for fluency and relevance. Temperature and sampling control randomness in word choice, affecting diversity and quality.

So, the key metrics are diversity (to avoid boring, repetitive text) and coherence (to keep text meaningful). We balance these by adjusting temperature and sampling methods.

Confusion Matrix or Equivalent Visualization

Unlike classification, temperature and sampling do not use confusion matrices. Instead, we visualize the probability distribution over next words.

    Example: Next word probabilities for "The cat sat on the"
    ---------------------------------------------
    Word       | Probability (Temp=1.0) | Probability (Temp=0.5)
    ---------------------------------------------
    mat        | 0.4                    | 0.7
    floor      | 0.3                    | 0.2
    roof       | 0.2                    | 0.05
    chair      | 0.1                    | 0.05
    ---------------------------------------------
    

Lower temperature sharpens the distribution, making the model pick more likely words. Higher temperature flattens it, increasing randomness.

Precision vs Recall Tradeoff Equivalent

In text generation, the tradeoff is between coherence and diversity.

• Low temperature (e.g., 0.2) means the model picks high-probability words, making text very coherent but repetitive and dull (low diversity).
• High temperature (e.g., 1.5) means the model picks words more randomly, increasing diversity but risking nonsense or off-topic text (low coherence).

Sampling methods like top-k or nucleus sampling help balance this by limiting choices to the most probable words, improving quality.

What "Good" vs "Bad" Looks Like

Good: Text that is fluent, relevant, and interesting. Diversity metrics show a healthy number of unique phrases without losing meaning. Temperature around 0.7 often works well.

Bad: Text that is repetitive, dull, or nonsensical. Too low temperature leads to repeated phrases. Too high temperature leads to gibberish or off-topic words.

Example:

• Low temp (0.1): "The cat sat on the mat. The cat sat on the mat." (boring repetition)
• High temp (1.5): "The cat sat on the galaxy banana elephant." (nonsense)

Common Pitfalls

• Ignoring diversity: Only looking at perplexity can hide repetitive text problems.
• Overusing high temperature: Leads to meaningless output, hurting user experience.
• Not tuning sampling: Using pure random sampling without limits can produce poor quality text.
• Misinterpreting metrics: High diversity is not always good if coherence is lost.

Self Check

Your language model generates text with a temperature of 1.2 and shows high diversity but many sentences are off-topic or confusing. Is this good for production?

Answer: No. While diversity is high, the coherence is low, making the text confusing. You should lower the temperature or adjust sampling to improve quality.

Key Result

Balancing temperature controls the tradeoff between coherent and diverse text generation.

Practice

1. What does increasing the temperature parameter in text generation usually do?

easy

A. Makes the output more predictable and repetitive

B. Stops the model from generating any text

C. Makes the output more random and creative

D. Always selects the most probable next word

Solution

1. Step 1: Understand temperature effect on randomness

   Temperature controls how much randomness is added to the word selection process in text generation.

2. Step 2: Relate temperature to creativity

   Higher temperature increases randomness, making the output more creative and less predictable.

3. Final Answer:

   Makes the output more random and creative -> Option C

4. Quick Check:

   Higher temperature = more randomness [OK]

Hint: Higher temperature means more randomness in output [OK]

Common Mistakes:

• Thinking higher temperature makes output more predictable
• Confusing temperature with model size
• Assuming temperature stops generation

2. Which of the following code snippets correctly applies temperature scaling to logits before sampling in Python?

easy

A. probs = softmax(logits / temperature)

B. probs = softmax(logits * temperature)

C. probs = softmax(logits + temperature)

D. probs = softmax(logits - temperature)

Solution

1. Step 1: Recall temperature scaling formula

   Temperature is applied by dividing logits by temperature before softmax to adjust randomness.

2. Step 2: Identify correct operation

   Dividing logits by temperature scales the logits correctly; multiplying or adding is incorrect.

3. Final Answer:

   probs = softmax(logits / temperature) -> Option A

4. Quick Check:

   Divide logits by temperature before softmax [OK]

Hint: Divide logits by temperature before softmax [OK]

Common Mistakes:

• Multiplying logits by temperature instead of dividing
• Adding temperature to logits
• Subtracting temperature from logits

3. Given logits = [2.0, 1.0, 0.1] and temperature = 0.5, what is the approximate probability of the first token after applying softmax with temperature scaling?

medium

A. About 0.30

B. About 0.60

C. About 0.50

D. About 0.84

Solution

1. Step 1: Scale logits by dividing by temperature

   Divide each logit by 0.5: [2.0/0.5=4.0, 1.0/0.5=2.0, 0.1/0.5=0.2]

2. Step 2: Calculate softmax probabilities

   Compute exp values: exp(4.0)=54.6, exp(2.0)=7.39, exp(0.2)=1.22; sum=63.21; probability first token = 54.6/63.21 ≈ 0.86 (approx 0.86 considering rounding)

3. Final Answer:

   About 0.86 -> Option D

4. Quick Check:

   Lower temperature sharpens distribution, first token ~0.86 [OK]

Hint: Divide logits by temperature, then softmax to find probabilities [OK]

Common Mistakes:

• Multiplying logits by temperature instead of dividing
• Skipping exponentiation step
• Using temperature incorrectly in softmax

4. A developer writes this code to sample a token with temperature 1.5 but always gets the same token. What is the likely bug?

scaled_logits = logits * temperature
probs = softmax(scaled_logits)
sampled_token = sample_from(probs)

medium

A. They should divide logits by temperature, not multiply

B. They forgot to apply softmax

C. Temperature should be zero to get randomness

D. Sampling function is incorrect

Solution

1. Step 1: Identify temperature scaling mistake

   The code multiplies logits by temperature, which is incorrect; it should divide logits by temperature.

2. Step 2: Explain effect of wrong scaling

   Multiplying by temperature >1 increases logits, making softmax peakier and less random, causing same token output.

3. Final Answer:

   They should divide logits by temperature, not multiply -> Option A

4. Quick Check:

   Divide logits by temperature for correct scaling [OK]

Hint: Divide, don't multiply logits by temperature [OK]

Common Mistakes:

• Multiplying instead of dividing logits
• Setting temperature to zero
• Ignoring softmax step

5. You want to generate text that balances creativity and coherence. Which temperature value and sampling strategy combination is best?

hard

A. Temperature 0.1 with greedy sampling

B. Temperature around 0.7 with top-k sampling

C. Temperature 2.0 with random sampling

D. Temperature 1.5 with no sampling (always pick max)

Solution

1. Step 1: Understand temperature impact on creativity

   Temperature ~0.7 balances randomness and predictability, avoiding too repetitive or too random output.

2. Step 2: Choose sampling method for balance

   Top-k sampling limits choices to top probable tokens, improving coherence while allowing creativity.

3. Final Answer:

   Temperature around 0.7 with top-k sampling -> Option B

4. Quick Check:

   Moderate temperature + top-k = balanced creativity [OK]

Hint: Use moderate temperature and top-k for balanced text [OK]

Common Mistakes:

• Using very low temperature causing boring text
• Using very high temperature causing nonsense
• Ignoring sampling method effects