Top-p Sampling in NLP: What It Is and How It Works

In NLP, top-p sampling is a method to generate text by selecting the next word from the smallest set of words whose combined probability is at least p. This helps models produce more diverse and natural outputs by focusing on the most likely words while ignoring very unlikely ones.

How It Works

Top-p sampling works by looking at the list of possible next words a model can choose, each with a probability score. Instead of always picking the single most likely word, it gathers the top words whose total probability adds up to a threshold p (like 0.9 or 90%).

Imagine you are picking candies from a jar, but you only want to pick from the most popular flavors that together make up 90% of all candies. You ignore the rare flavors that make up the last 10%. This way, you keep choices diverse but still sensible.

By sampling from this smaller group, the model avoids boring or repetitive text and can create more interesting and varied sentences.

Example

This example shows how to apply top-p sampling to pick the next word from a list of word probabilities.

python

import numpy as np

def top_p_sampling(probabilities, p=0.9):
    sorted_indices = np.argsort(probabilities)[::-1]
    sorted_probs = probabilities[sorted_indices]
    cumulative_probs = np.cumsum(sorted_probs)
    cutoff = np.where(cumulative_probs >= p)[0][0] + 1
    filtered_indices = sorted_indices[:cutoff]
    filtered_probs = sorted_probs[:cutoff]
    filtered_probs /= filtered_probs.sum()  # normalize
    chosen_index = np.random.choice(filtered_indices, p=filtered_probs)
    return chosen_index

# Example probabilities for 5 words
probs = np.array([0.4, 0.3, 0.15, 0.1, 0.05])
np.random.seed(42)  # for reproducibility
chosen = top_p_sampling(probs, p=0.8)
print(f"Chosen word index: {chosen}")

Output

Chosen word index: 1

When to Use

Top-p sampling is useful when you want your NLP model to generate creative, varied, and natural-sounding text. It balances between always picking the most likely word (which can be repetitive) and picking completely random words (which can be nonsensical).

Common use cases include chatbots, story generation, and any text generation task where diversity and fluency matter. It helps avoid dull or repetitive responses while keeping the output sensible.

Key Points

• Top-p sampling selects words from the smallest group with cumulative probability ≥ p.
• It improves text diversity compared to always picking the highest probability word.
• It avoids unlikely words that could make text nonsensical.
• It is widely used in modern NLP text generation tasks.

Key Takeaways

Top-p sampling picks the next word from a group covering the top cumulative probability p.

It balances diversity and coherence in generated text.

It is better than always choosing the most likely word or random sampling.

Use it for creative and natural text generation like chatbots and stories.