Positional Encoding: Definition and Examples

Positional Encoding is a fundamental mechanism of the Transformer architecture, introduced in the paper "Attention Is All You Need" (2017). Unlike recurrent networks (RNNs), which process tokens sequentially and naturally capture order, Transformers process all tokens in parallel via the attention mechanism. Without positional information, the model would make no distinction between "The cat eats the mouse" and "The mouse eats the cat."

The original positional encoding uses sinusoidal functions of different frequencies to generate a unique vector for each position in the sequence. These vectors are added to token embeddings before entering the Transformer. The sine and cosine functions allow the model to generalize to sequence lengths not seen during training and to capture relative distance relationships between tokens.

Since the foundational paper, several variants have emerged. Learned positional embeddings let the model directly learn position vectors during training, as in BERT and GPT. More recently, RoPE (Rotary Position Embedding), used in LLaMA and other modern models, encodes position via rotations in vector space, offering better extrapolation to long sequences.

For language model users, Positional Encoding has a direct impact: it determines the model's context window (4K, 128K, 1M tokens, etc.) and the model's ability to maintain coherence over long texts. Context extension techniques like ALiBi or position interpolation allow extending this window beyond what was seen during training.