Cross Attention: Definition and Examples

Cross Attention is a fundamental mechanism in Transformer architectures that allows two distinct sequences to interact with each other. Unlike Self Attention, where a sequence attends to itself, Cross Attention uses one sequence as the source of queries and another as the source of keys and values. This allows the model to determine which parts of one sequence are relevant for each element of the other.

This mechanism was introduced in the original Transformer architecture of 2017 ("Attention Is All You Need") within the decoder, where the sequence being generated queries the encoded input sequence. For example, during machine translation, each word generated in the target language can "look at" the entire source sentence to determine which words are most relevant at that stage of translation.

Cross Attention has become an essential component of multimodal models. In image generation systems like Stable Diffusion, it allows descriptive text (the prompt) to precisely guide visual creation: each region of the image can rely on the most relevant words of the prompt. This mechanism explains why the formulation of a prompt so strongly influences the generated visual result.

In practice, Cross Attention works in three steps: the first sequence is projected into query vectors (Q), the second into key vectors (K) and value vectors (V). An attention score is computed between each query and each key, then normalized by softmax. These scores weight the values to produce the output. This mechanism is differentiable and end-to-end trainable, allowing the model to automatically learn which cross-sequence correspondences are useful for the task.