Grouped Query Attention: Definition and Examples

Grouped Query Attention (GQA) is a Transformer architecture optimization technique introduced by Google researchers in 2023. It sits halfway between classic Multi-Head Attention (MHA), where each query head has its own keys and values, and Multi-Query Attention (MQA), where all heads share a single set of key-value pairs. In GQA, query heads are divided into groups, and each group shares the same set of keys and values.

The main benefit of GQA is the significant reduction in memory required to store the KV cache during inference. In a classic model with 32 attention heads, the KV cache must store 32 key-value pairs per layer. With GQA using 8 groups, this number drops to 8, reducing KV cache memory by 75%, while maintaining generation quality very close to MHA.

This technique has become a standard in modern language models. Meta's Llama 2 (70B) was one of the first large models to adopt GQA, followed by Mistral, Llama 3, and many others. GQA enables these models to handle longer contexts and serve more simultaneous requests with the same hardware, which is crucial for production deployment.

In practice, GQA speeds up the decoding phase (token-by-token generation) without significantly degrading response quality. Benchmarks show that performance loss is generally below 1% compared to full MHA, while inference speed gains range from 30% to 50% depending on the configuration. This favorable trade-off explains its widespread adoption.