Attention: Definition and Examples

Attention is a mechanism introduced in the seminal paper 'Attention Is All You Need' (2017) by the Google team. It is the fundamental building block of the Transformer architecture, on which all current large language models such as GPT, Claude, or Gemini are based. The principle: rather than processing words of a sentence sequentially and uniformly, the model learns to 'look at' all words simultaneously and assign different weights to each according to its relevance to the task at hand.

Concretely, the attention mechanism works with three vectors — Query, Key, and Value — computed for each token in the sequence. The attention score between two tokens is obtained by the dot product of their Query and Key vectors, then normalized. This score determines how much one word should 'pay attention' to another. For example, in the sentence 'The cat sleeps on the couch', the word 'sleeps' will assign a high weight to 'cat' because it is the subject of the action.

The most widely used variant is 'self-attention', where each token computes its attention scores with respect to all other tokens in the same sequence. Transformers also use 'multi-head attention', which runs several attention mechanisms in parallel, allowing the model to capture different types of relationships (syntactic, semantic, logical) simultaneously.

In prompt engineering, understanding attention is crucial because it explains why the position and wording of instructions in a prompt directly influence the quality of responses. Information placed at the beginning and end of the prompt generally receives more attention, and clear, structured instructions help the attention mechanism identify what is relevant.