Sliding Window Attention: Definition and Examples

Sliding Window Attention is an optimized variant of the Transformer attention mechanism. Unlike full attention, where each token interacts with every other token in the sequence, Sliding Window Attention restricts each token's attention field to a fixed window of immediate neighbors. For example, with a window size of 4096, each token only "looks at" the previous 4096 tokens.

This approach solves a fundamental problem of Transformers: the quadratic complexity of full attention. When a model needs to process a sequence of 100,000 tokens, computing attention between all token pairs becomes extremely expensive in memory and computation. Sliding Window Attention reduces this to linear complexity with respect to sequence length, making it possible to handle very long contexts.

A subtle but essential aspect is the propagation effect across layers. Even though each layer only sees a local window, stacking multiple layers allows information to propagate over much longer distances. With L layers and a window size of W, information can theoretically traverse L × W tokens, providing an effective range far larger than the individual window size.

This mechanism was popularized by models like Mistral 7B and Mixtral, and is a key component of the Longformer architecture. It is often combined with other techniques such as global attention on special tokens or Grouped Query Attention (GQA) to achieve an optimal balance between performance and efficiency.