Temperature and sampling in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the role of temperature in sampling from a language model?

Temperature controls how random or focused the model's predictions are. A low temperature (<1) makes the model more confident and conservative, picking high-probability words. A high temperature (>1) makes the model more creative and random by flattening the probabilities.

beginner

Explain sampling in the context of generating text from a language model.

Sampling means picking the next word based on the model's predicted probabilities instead of always choosing the most likely word. This adds variety and creativity to the generated text.

intermediate

How does increasing temperature affect the probability distribution during sampling?

Increasing temperature makes the probability distribution more even, so less likely words have a higher chance to be picked. This leads to more diverse and surprising outputs.

intermediate

What happens if temperature is set to 0 during sampling?

Setting temperature to 0 means always picking the word with the highest probability (greedy decoding). This removes randomness and can make the output repetitive or dull.

beginner

Why might you want to use a moderate temperature (e.g., 0.7) instead of very low or very high?

A moderate temperature balances creativity and coherence. It allows some randomness for interesting text but keeps the output sensible and relevant.

What does a temperature of 1.0 mean when sampling from a language model?

A Probabilities are flattened to be more even

B The original predicted probabilities are used without change

C Only the highest probability word is chosen

D Probabilities are sharpened to favor the top word

A temperature of 1.0 means the model uses the original predicted probabilities as they are, without scaling.

What is the effect of setting temperature to a very high value (e.g., 5)?

A Model always picks the most likely word

B Output becomes very predictable and repetitive

C Model ignores probabilities and picks words randomly

D Output becomes more random and diverse

High temperature flattens the probability distribution, increasing randomness and diversity in output.

Which sampling method removes randomness completely?

A Greedy decoding (temperature = 0)

B Random sampling without temperature

C Sampling with temperature = 1

D Sampling with temperature > 1

Greedy decoding always picks the highest probability word, removing randomness.

Why is sampling preferred over always picking the highest probability word?

A It makes the output more creative and less repetitive

B It guarantees the most accurate output

C It speeds up the generation process

D It reduces the model size

Sampling adds randomness, making generated text more varied and interesting.

What does lowering temperature below 1 do to the output?

A Makes output more random

B Makes output longer

C Makes output more focused and conservative

D Makes output shorter

Lower temperature sharpens the probability distribution, making the model pick more likely words.

Describe how temperature affects the randomness of text generated by a language model.

Think about how temperature changes the chance of picking less likely words.

Explain why sampling is used instead of always choosing the most likely word in text generation.

Consider how always picking the top word might affect the text.

Practice

1. What does increasing the temperature parameter in text generation usually do?

easy

A. Makes the output more predictable and repetitive

B. Stops the model from generating any text

C. Makes the output more random and creative

D. Always selects the most probable next word

Solution

1. Step 1: Understand temperature effect on randomness

   Temperature controls how much randomness is added to the word selection process in text generation.

2. Step 2: Relate temperature to creativity

   Higher temperature increases randomness, making the output more creative and less predictable.

3. Final Answer:

   Makes the output more random and creative -> Option C

4. Quick Check:

   Higher temperature = more randomness [OK]

Hint: Higher temperature means more randomness in output [OK]

Common Mistakes:

• Thinking higher temperature makes output more predictable
• Confusing temperature with model size
• Assuming temperature stops generation

2. Which of the following code snippets correctly applies temperature scaling to logits before sampling in Python?

easy

A. probs = softmax(logits / temperature)

B. probs = softmax(logits * temperature)

C. probs = softmax(logits + temperature)

D. probs = softmax(logits - temperature)

Solution

1. Step 1: Recall temperature scaling formula

   Temperature is applied by dividing logits by temperature before softmax to adjust randomness.

2. Step 2: Identify correct operation

   Dividing logits by temperature scales the logits correctly; multiplying or adding is incorrect.

3. Final Answer:

   probs = softmax(logits / temperature) -> Option A

4. Quick Check:

   Divide logits by temperature before softmax [OK]

Hint: Divide logits by temperature before softmax [OK]

Common Mistakes:

• Multiplying logits by temperature instead of dividing
• Adding temperature to logits
• Subtracting temperature from logits

3. Given logits = [2.0, 1.0, 0.1] and temperature = 0.5, what is the approximate probability of the first token after applying softmax with temperature scaling?

medium

A. About 0.30

B. About 0.60

C. About 0.50

D. About 0.84

Solution

1. Step 1: Scale logits by dividing by temperature

   Divide each logit by 0.5: [2.0/0.5=4.0, 1.0/0.5=2.0, 0.1/0.5=0.2]

2. Step 2: Calculate softmax probabilities

   Compute exp values: exp(4.0)=54.6, exp(2.0)=7.39, exp(0.2)=1.22; sum=63.21; probability first token = 54.6/63.21 ≈ 0.86 (approx 0.86 considering rounding)

3. Final Answer:

   About 0.86 -> Option D

4. Quick Check:

   Lower temperature sharpens distribution, first token ~0.86 [OK]

Hint: Divide logits by temperature, then softmax to find probabilities [OK]

Common Mistakes:

• Multiplying logits by temperature instead of dividing
• Skipping exponentiation step
• Using temperature incorrectly in softmax

4. A developer writes this code to sample a token with temperature 1.5 but always gets the same token. What is the likely bug?

scaled_logits = logits * temperature
probs = softmax(scaled_logits)
sampled_token = sample_from(probs)

medium

A. They should divide logits by temperature, not multiply

B. They forgot to apply softmax

C. Temperature should be zero to get randomness

D. Sampling function is incorrect

Solution

1. Step 1: Identify temperature scaling mistake

   The code multiplies logits by temperature, which is incorrect; it should divide logits by temperature.

2. Step 2: Explain effect of wrong scaling

   Multiplying by temperature >1 increases logits, making softmax peakier and less random, causing same token output.

3. Final Answer:

   They should divide logits by temperature, not multiply -> Option A

4. Quick Check:

   Divide logits by temperature for correct scaling [OK]

Hint: Divide, don't multiply logits by temperature [OK]

Common Mistakes:

• Multiplying instead of dividing logits
• Setting temperature to zero
• Ignoring softmax step

5. You want to generate text that balances creativity and coherence. Which temperature value and sampling strategy combination is best?

hard

A. Temperature 0.1 with greedy sampling

B. Temperature around 0.7 with top-k sampling

C. Temperature 2.0 with random sampling

D. Temperature 1.5 with no sampling (always pick max)

Solution

1. Step 1: Understand temperature impact on creativity

   Temperature ~0.7 balances randomness and predictability, avoiding too repetitive or too random output.

2. Step 2: Choose sampling method for balance

   Top-k sampling limits choices to top probable tokens, improving coherence while allowing creativity.

3. Final Answer:

   Temperature around 0.7 with top-k sampling -> Option B

4. Quick Check:

   Moderate temperature + top-k = balanced creativity [OK]

Hint: Use moderate temperature and top-k for balanced text [OK]

Common Mistakes:

• Using very low temperature causing boring text
• Using very high temperature causing nonsense
• Ignoring sampling method effects