Presence Penalty: Definition and Examples

The Presence Penalty is a hyperparameter available in most language model APIs such as those from OpenAI or Cohere. Its role is simple: each time a token has already been generated in the response, a fixed penalty is applied to its probability of being selected again. Unlike Frequency Penalty, which increases proportionally with the number of occurrences, Presence Penalty applies in a binary manner — the token has either already appeared (penalized) or never appeared (not penalized).

Concretely, this parameter generally accepts a value between -2.0 and 2.0. A positive value discourages repetition and pushes the model to explore new words and concepts. A negative value, conversely, encourages the model to reuse already mentioned terms, which can reinforce thematic coherence. A value of 0 completely disables the effect.

The main benefit of Presence Penalty lies in its ability to control the lexical and thematic diversity of responses. When increased, the model tends to address more different topics and vary its vocabulary. This is particularly useful for brainstorming, creative generation, or any task requiring broad coverage of concepts.

It is important to distinguish Presence Penalty from Frequency Penalty and Temperature, which all influence output diversity but through different mechanisms. A good prompt engineer knows how to combine these parameters in a balanced way to obtain the desired model behavior, without sacrificing coherence for diversity, or vice versa.