KV Cache: Definition and Examples

KV Cache, or key-value cache, is a fundamental optimization technique in the operation of large language models based on the Transformer architecture. During text generation, the attention mechanism must compute key vectors (Keys) and value vectors (Values) for each token in the sequence. Without caching, these computations would be fully repeated at every new token produced, which would be extremely resource-intensive.

The principle of KV Cache is simple: once a token has been processed, its key and value vectors are stored in memory. When the model generates the next token, it only needs to compute K and V for that new token, then combines them with the already cached values. This transforms a quadratic operation into a nearly linear one, significantly speeding up generation.

In practice, KV Cache determines the amount of GPU memory required to serve a model. For a model like GPT-4 or Claude, the cache can consume several gigabytes of VRAM per request, especially with long contexts. This is why techniques like Multi-Query Attention (MQA), Grouped-Query Attention (GQA), or cache quantization have been developed to reduce this memory footprint.

For the end user, KV Cache explains why the first token of a response is slower to appear (prefill phase where the entire prompt is processed) while subsequent tokens arrive much faster (decode phase where the cache is reused). It also explains why requests with very long contexts are more expensive and slower.