Speculative Decoding: Definition and Examples

Speculative Decoding is an optimization method designed to accelerate text generation by large language models. The principle is based on a simple observation: standard autoregressive generation, where each token is produced one by one, massively underutilizes GPU hardware because each step requires a full forward pass through the model. Speculative decoding bypasses this bottleneck by introducing a "draft then verify" mechanism.

Concretely, a smaller, faster auxiliary model (called the draft model) speculatively generates a sequence of candidate tokens. Then, the main model (the target model, larger and more accurate) verifies all these tokens in a single forward pass, in parallel. Accepted tokens are kept, and generation resumes from the first rejected token. This verification leverages the fact that evaluating multiple tokens simultaneously costs almost as much as evaluating a single token on a modern GPU.

The mathematical elegance of this approach lies in its guarantee: thanks to a carefully designed acceptance-rejection scheme, the final probability distribution is exactly identical to that of the target model alone. In other words, Speculative Decoding produces results that are strictly equivalent in quality, but 2 to 3 times faster. The gain depends on the acceptance rate, i.e., the ability of the small model to correctly predict what the large model would have generated.

This technique has become a standard in production systems of major AI providers. Variants exist, such as self-speculative decoding (where the model uses its own intermediate layers as a draft), Medusa (which adds parallel prediction heads), or tree-based speculative decoding, which explores multiple possible continuations simultaneously to maximize the acceptance rate.