Tokenization: Definition and Examples

Tokenization is the fundamental step by which an artificial intelligence model transforms raw text into a sequence of tokens, i.e., numerical units it can process. Without this step, an LLM would be unable to understand or generate human language.

Contrary to what one might think, a token does not always correspond to a whole word. Modern tokenization algorithms like BPE (Byte Pair Encoding) or SentencePiece split text into frequent sub-units. For example, the word 'unbelievably' might be split into 'unbelievably' + 'ably'. Common words like 'the' or 'is' usually form a single token, while rare or technical words are fragmented into several tokens.

This mechanism has direct consequences for using LLMs. The number of tokens determines the cost of an API call, the maximum length of a conversation (context window), and even the quality of responses. As a rule of thumb, a token represents about 3 to 4 characters in French, or about 0.75 words. French thus consumes slightly more tokens than English to express the same idea.

Understanding tokenization allows one to optimize prompts: reduce unnecessary tokens, anticipate context limits, and better estimate costs. It is a key skill for any prompt engineering practitioner who wants to work efficiently with language model APIs.