Byte Pair Encoding: Definition and Examples

Byte Pair Encoding, or BPE, is an algorithm originally designed for data compression by Philip Gage in 1994. Its principle is simple: it identifies the most frequent adjacent symbol pairs in a text corpus and merges them iteratively to create new units. This approach was brilliantly repurposed in the field of natural language processing (NLP) to solve the fundamental problem of tokenization.

In practice, BPE builds a fixed-size vocabulary from a training corpus. The algorithm starts with a vocabulary consisting of all individual characters, then progressively merges the most frequent pairs until the desired vocabulary size is reached. For example, the letters "e" and "s" often appearing together will be merged into a token "es", which can then be further merged with others.

This method is at the core of large language models such as GPT, Claude, and LLaMA. It offers an optimal balance between a character-level vocabulary (too granular and computationally expensive) and a word-level vocabulary (too rigid for rare or unknown words). Thanks to BPE, a model can handle any word, even one never seen during training, by breaking it down into known sub-units.

Understanding BPE is essential for prompt engineering practitioners because how a text is split into tokens directly influences context limits, API call costs, and sometimes even the quality of the model's responses.