Top K: Definition and Examples

Top K is a sampling parameter used during text generation by language models (LLMs). At each generation step, the model calculates a probability for each possible token in its vocabulary. The Top K parameter restricts this choice to the K tokens with the highest probabilities, excluding all others from selection.

Concretely, if Top K is set to 50, the model will only consider the 50 most probable tokens for the next word, then perform a random draw among these 50 candidates according to their respective probabilities. A Top K of 1 always selects the most probable token (deterministic generation), while a very high Top K allows the model to choose from a wide range of possibilities.

The main benefit of Top K is finding a balance between coherence and creativity. A value too low produces repetitive and predictable texts, while a value too high may introduce improbable tokens that make the text incoherent. In practice, Top K is often used in combination with other parameters like Temperature and Top P to fine-tune model behavior.

It is important to note that Top K has a limitation: it applies a fixed threshold regardless of context. In some situations, the model is very confident and only 5 tokens are truly relevant, while in other cases, 200 tokens could be valid continuations. This is why Top P (nucleus sampling) is often preferred, as it dynamically adapts to the probability distribution.