Self Attention: Definition and Examples

Self Attention (or auto-attention) is a fundamental mechanism in artificial intelligence that allows a language model to analyze relationships between all words in the same sequence. Unlike recurrent networks that process words one by one from left to right, Self Attention enables each word to "look at" all other words in the sentence simultaneously to better understand context.

Concretely, for each word in the sequence, the mechanism computes three vectors: a Query (what the word is looking for), a Key (what the word offers as information), and a Value (the actual information it carries). By comparing a word's Query with the Keys of all other words, the model determines an attention score indicating how relevant each word is for understanding the current word.

This mechanism is at the core of the Transformer architecture, introduced by Google in 2017 in the paper "Attention Is All You Need". Models like GPT, Claude, or Gemini stack dozens of Self Attention layers, allowing them to capture complex dependencies between words, even far apart in the text. It is thanks to Self Attention that a model can understand that in the sentence "The cat that was sleeping on the living room couch got up," the verb "got up" refers to "cat" despite the distance.

For prompt engineering practitioners, understanding Self Attention helps explain why models excel in certain tasks (summarization, translation, context analysis) but can also be sensitive to context length and the position of key information in a prompt.