Frequency Penalty: Definition and Examples

The Frequency Penalty is a hyperparameter available in language model APIs such as OpenAI's GPT. It applies an increasing penalty to each token based on how many times that token has already been used in the generated response. The more frequently a word appears, the less likely it is to be selected again.

Concretely, this parameter works by subtracting a value proportional to the number of occurrences of a token from the logits (probability scores) before sampling. For example, with a Frequency Penalty of 0.5, a word that has appeared 3 times will have its score reduced by 1.5 (3 × 0.5). This penalty is linear: it increases with each new occurrence, unlike the Presence Penalty which applies a fixed penalty upon first appearance.

The parameter typically accepts values between 0 and 2. At 0, no penalty is applied and the model can freely repeat the same words. Moderate values (0.3-0.8) reduce repetitions while keeping the text natural. High values (above 1.5) force the model to use very varied vocabulary, risking artificial or incoherent phrasing.

In prompt engineering, the Frequency Penalty is particularly useful for creative content generation, article writing, or any task where lexical diversity is desirable. It differs from the Temperature parameter, which controls overall randomness, by specifically targeting the issue of word repetition.