Tokens (AI): Definition and Examples
Tokens are the elementary units of text that AI models use to read and generate content. A token corresponds on average to 3/4 of a word in English and 1/2 word in French.
Full definition
When you send a message to a language model, your text is first split into tokens. A token is not necessarily a whole word: it can be a short word, a part of a word, or a punctuation mark.
This tokenization is essential because models work with numerical sequences. The vocabulary (50K-100K tokens) is fixed during training.
Tokens impact cost and limits. Each model has a maximum context size in tokens (GPT-4 = 128K, Claude = 200K).
In French, count 1.5 to 2 tokens per word, compared to 1-1.3 in English. Accents and long words consume more tokens.
Etymology
The term comes from computing where it denotes a lexical unit. Modern techniques (BPE, SentencePiece) date from the 2010s-2020s.
Concrete examples
Understanding token counting
How many tokens does this text contain? Explain the token-by-token breakdown.
Optimizing a prompt to save tokens
Summarize this document in 500 words max. Priority: numbers, dates, decisions.
Estimating the cost of an API call
Practical usage
Keep your prompts concise. Use structured instructions. Count about 750 tokens for 1000 characters in French.
Related concepts
FAQ
How to count tokens?
Why does French consume more tokens?
What happens if you exceed the limit?
See also
How to use this prompt
- Copy the prompt with the button above.
- Paste it into ChatGPT, Claude or your favorite AI assistant.
- Replace the bracketed variables with your details, then refine the result.
About Prompt Guide
Prompt Guide is a free library of 2500+ ready-to-use prompts for ChatGPT, Claude and other AIs, with guides to learn prompting and tools to build and optimize your own prompts.
More definitions
Tool Calling: Definition and Examples
Tool Calling is the ability of a language model to identify when it should use an external tool and to generate the structured parameters
Tool Use: Definition and Examples
Tool Use (or function calling) is the ability of a language model to interact with external tools — APIs, databases, calculators, browsers
Top K: Definition and Examples
Top K is a generation parameter that limits the model's choice to the K most probable tokens at each step, reducing incoherent responses.
Top P: Definition and Examples
Top P, also known as nucleus sampling, is a generation parameter that controls the diversity of AI responses by limiting token selection to those with cumulative probability reaching a threshold P.
Transfer Learning: Definition and Examples
Transfer learning is a machine learning technique that reuses a pre-trained model on one task to adapt it to a new one.
Transformer: Definition and Examples
Neural network architecture introduced in 2017 by Google, based on the attention mechanism, which forms the basis of all major language models.
Get new prompts every week
Join our newsletter.