Top P: Definition and Examples

Top P (or nucleus sampling) is a sampling method used during text generation by language models. Rather than considering the entire vocabulary at each generation step, the model only retains the most probable tokens whose cumulative probabilities reach the defined threshold P. For example, with a Top P of 0.9, the model selects the most probable tokens until their sum of probabilities reaches 90%, then randomly chooses from this subset.

This mechanism offers a major advantage over Top K: it dynamically adapts to context. If the model is very confident about the next word (e.g., after "the Eiffel"), the nucleus will be small and contain only a few candidates like "Tower" or "of". Conversely, in a more open context (e.g., the beginning of a story), the nucleus naturally expands to include more creative possibilities.

Top P is set between 0 and 1. A value close to 0 makes the model almost deterministic by keeping only the most probable token(s). A value of 1 disables filtering and considers the entire vocabulary. Common values range between 0.7 and 0.95 depending on use: lower for factual tasks, higher for creative tasks.

It is important to note that Top P interacts with temperature. Temperature modifies the probability distribution before Top P filtering. In practice, it is recommended to adjust one or the other, rather than both simultaneously, to maintain predictable control over model behavior.