Tokens (AI): Definition and Examples
Tokens are the elementary units of text that AI models use to read and generate content. A token corresponds on average to 3/4 of a word in English and 1/2 word in French.
Full definition
When you send a message to a language model, your text is first split into tokens. A token is not necessarily a whole word: it can be a short word, a part of a word, or a punctuation mark.
This tokenization is essential because models work with numerical sequences. The vocabulary (50K-100K tokens) is fixed during training.
Tokens impact cost and limits. Each model has a maximum context size in tokens (GPT-4 = 128K, Claude = 200K).
In French, count 1.5 to 2 tokens per word, compared to 1-1.3 in English. Accents and long words consume more tokens.
Etymology
The term comes from computing where it denotes a lexical unit. Modern techniques (BPE, SentencePiece) date from the 2010s-2020s.
Concrete examples
Understanding token counting
How many tokens does this text contain? Explain the token-by-token breakdown.
Optimizing a prompt to save tokens
Summarize this document in 500 words max. Priority: numbers, dates, decisions.
Estimating the cost of an API call
Practical usage
Keep your prompts concise. Use structured instructions. Count about 750 tokens for 1000 characters in French.
Related concepts
FAQ
How to count tokens?
Why does French consume more tokens?
What happens if you exceed the limit?
See also
How to use this prompt
- Copy the prompt with the button above.
- Paste it into ChatGPT, Claude or your favorite AI assistant.
- Replace the bracketed variables with your details, then refine the result.
About Prompt Guide
Prompt Guide is a free library of 2500+ ready-to-use prompts for ChatGPT, Claude and other AIs, with guides to learn prompting and tools to build and optimize your own prompts.
More definitions
Trustworthy AI: Definition and Examples
Trustworthy AI refers to artificial intelligence designed to be reliable, ethical, transparent, and respectful of fundamental rights.
Video Understanding: Definition and Examples
Ability of an AI model to analyze, interpret, and extract relevant information from video content, combining visual, temporal, and often audio understanding.
Vision RAG: Definition and Examples
Vision RAG is an extension of Retrieval-Augmented Generation that integrates visual documents (images, charts, scanned PDFs) into the search process.
World Model: Definition and Examples
A world model is an internal representation that an AI system builds of the external world, allowing it to simulate, predict, and reason about the consequences of its actions without having to execute them in reality.
Zero-Shot Prompting: Definition and Examples
Zero-shot prompting gives the AI an instruction without any examples. Discover when and how to use this technique.
A2A Agent To Agent: Definition and Examples
A2A (Agent-to-Agent) is an open protocol developed by Google that allows autonomous AI agents to communicate, collaborate, and delegate tasks between each other.
Get new prompts every week
Join our newsletter.