Rotary Position Embedding: Definition and Examples

Rotary Position Embedding, or RoPE, is a positional encoding method introduced by Jianlin Su et al. in 2021. Unlike classic positional encodings (sinusoidal or learned), RoPE encodes each token's position by applying a geometric rotation to the query and key vectors in the attention mechanism. This rotation causes the dot product between two vectors to naturally depend on their relative distance, without the need to explicitly add a positional bias.

The fundamental idea is based on complex numbers and rotations in a two-dimensional space. Each consecutive pair of dimensions of the embedding vector is treated as a complex number, then multiplied by a rotation factor whose angle depends on the token's position. Thus, the further apart two tokens are in the sequence, the greater the relative rotation between their representations, allowing the model to perceive the distance between words.

RoPE has several major advantages: it elegantly provides relative positional encoding, it is compatible with linear attention mechanisms, and it offers better generalization to sequence lengths not seen during training. The latter property has been particularly exploited with techniques like YaRN or NTK-aware scaling, which allow extending the model's context window.

Today, RoPE has become the de facto standard for modern large language models. It is used in LLaMA, Mistral, Qwen, PaLM, and many other models. Its ability to handle long contexts (up to millions of tokens with proper extensions) makes it a cornerstone of current LLM architectures.