Top-p and top-k sampling in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Top-k Sampling

In top-k sampling, what does the parameter k control when generating text from a language model?

A The maximum length of the generated text sequence

B The number of highest probability tokens considered for sampling at each step

C The temperature scaling factor applied to logits before sampling

D The cumulative probability threshold to include tokens for sampling

💡 Hint

Think about how many tokens the model looks at before picking the next word.

Top-k sampling limits the candidate tokens to the k most probable ones at each step. This means the model only picks from these top k tokens, ignoring the rest.

🧠 Conceptual

intermediate

Understanding Top-p (Nucleus) Sampling

What does the parameter p represent in top-p (nucleus) sampling?

A The maximum number of tokens generated

B The temperature value to adjust randomness

C The fixed number of tokens to consider for sampling

D The cumulative probability threshold to include tokens for sampling

💡 Hint

It relates to the total probability mass of tokens considered.

Top-p sampling selects the smallest set of tokens whose cumulative probability exceeds p. This means it dynamically chooses tokens based on their combined probability.

❓ Predict Output

advanced

Output of Top-k Sampling Code Snippet

What is the output of the following Python code simulating top-k sampling probabilities?

Prompt Engineering / GenAI

import numpy as np
np.random.seed(0)
logits = np.array([0.1, 0.2, 0.3, 0.4, 0.5])
k = 3
# Select top-k logits
indices = np.argsort(logits)[-k:]
topk_probs = np.exp(logits[indices]) / np.sum(np.exp(logits[indices]))
sampled_index = np.random.choice(indices, p=topk_probs)
print(sampled_index)

A 3

B 0

C 4

D 2

💡 Hint

Check which indices are top 3 and how probabilities are computed.

The top 3 logits are at indices 2, 3, and 4 with values 0.3, 0.4, and 0.5. After softmax, index 4 has the highest probability. With the given seed, the sampled index is 4.

❓ Metrics

advanced

Effect of Top-p on Diversity Metrics

If you decrease the top-p value from 0.9 to 0.5 during text generation, what is the expected effect on the diversity of generated text?

A Diversity decreases because fewer tokens are considered

B Diversity increases because more tokens are considered

C Diversity remains the same because top-p does not affect token selection

D Diversity fluctuates randomly regardless of top-p

💡 Hint

Think about how cumulative probability threshold limits token choices.

Lowering top-p reduces the set of tokens considered, thus reducing diversity since the model picks from fewer options.

🔧 Debug

expert

Identifying Error in Top-k Sampling Implementation

Consider this code snippet for top-k sampling. Which option correctly identifies the error causing incorrect sampling?

Prompt Engineering / GenAI

import numpy as np
logits = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
k = 2
indices = np.argsort(logits)[:k]
topk_logits = logits[indices]
probs = np.exp(topk_logits) / np.sum(np.exp(topk_logits))
sampled_index = np.random.choice(indices, p=probs)
print(sampled_index)

A The code selects the lowest k logits instead of the highest k logits

B The softmax calculation is incorrect because it should use log probabilities

C The sampling should be done over logits, not indices

D The random seed is missing, causing non-reproducible results

💡 Hint

Check how argsort is used to select top-k logits.

Using np.argsort(logits)[:k] selects the smallest k logits, not the largest. It should use [-k:] to get the top k highest logits.