Temperature and sampling in NLP

Introduction

Temperature and sampling help control how creative or random a language model's text predictions are.

When you want the model to give more varied and creative text instead of always the most likely word.

When generating stories or poems where surprise and diversity are good.

When you want to balance between safe and risky word choices in chatbot replies.

When exploring different possible next words to understand model behavior.

When tuning text generation to be more predictable or more random.

Syntax

NLP

def sample_with_temperature(logits, temperature=1.0):
    import numpy as np
    logits = logits / temperature
    exp_logits = np.exp(logits - np.max(logits))
    probs = exp_logits / np.sum(exp_logits)
    return np.random.choice(len(probs), p=probs)

Temperature is a positive number; lower than 1 makes output more focused, higher than 1 makes it more random.

Sampling means picking the next word based on probabilities, not just the highest one.

Examples

Lower temperature makes the model pick more confident words, less random.

NLP

sample_with_temperature(logits, temperature=0.5)

Temperature 1 means normal sampling from the original probabilities.

NLP

sample_with_temperature(logits, temperature=1.0)

Higher temperature makes the model pick more surprising or rare words.

NLP

sample_with_temperature(logits, temperature=2.0)

Sample Model

This code shows how changing temperature affects which index (word) is picked from logits representing word scores.

NLP

import numpy as np

def sample_with_temperature(logits, temperature=1.0):
    logits = logits / temperature
    exp_logits = np.exp(logits - np.max(logits))
    probs = exp_logits / np.sum(exp_logits)
    return np.random.choice(len(probs), p=probs)

# Example logits for 5 possible next words
logits = np.array([2.0, 1.0, 0.1, 0.5, 0.0])

print('Sampling with temperature 0.5:')
for _ in range(5):
    idx = sample_with_temperature(logits, temperature=0.5)
    print(f'Chosen index: {idx}')

print('\nSampling with temperature 1.0:')
for _ in range(5):
    idx = sample_with_temperature(logits, temperature=1.0)
    print(f'Chosen index: {idx}')

print('\nSampling with temperature 2.0:')
for _ in range(5):
    idx = sample_with_temperature(logits, temperature=2.0)
    print(f'Chosen index: {idx}')

Important Notes

Temperature close to zero makes the model almost always pick the highest scoring word.

Sampling with temperature helps avoid boring or repetitive text.

Try different temperatures to find the best balance for your task.

Summary

Temperature controls randomness in text generation.

Lower temperature = more predictable, higher temperature = more creative.

Sampling picks words based on adjusted probabilities, not just the top choice.