Autoregressive Model: Definition and Examples

An autoregressive model works on a simple but powerful principle: it generates a sequence element by element, using each already produced element to predict the next. In the case of large language models (LLMs) like GPT or Claude, this means the model predicts the next token (word or subword) based on all previous tokens.

Concretely, when you ask an AI chatbot a question, the model does not generate its answer all at once. It produces a first word, then uses that word (and your question) to choose the second word, then uses the first two words to choose the third, and so on. This is why you often see the text appear progressively, word by word.

This sequential approach has important implications. On one hand, it allows the model to produce coherent and contextually relevant texts, because each new token takes into account everything that precedes it. On the other hand, it introduces latency proportional to the length of the response, since each token requires a full pass through the neural network.

Autoregressive models are opposed to so-called 'non-autoregressive' models that attempt to generate all elements simultaneously, often sacrificing quality for speed. Almost all current LLMs (GPT-4, Claude, Gemini, LLaMA) are autoregressive, making them the dominant architecture in generative text AI.