Scaling Laws: Definition and Examples

Scaling laws are among the most fundamental discoveries in modern AI research. Formalized notably by OpenAI researchers (Kaplan et al., 2020) and later refined by the DeepMind team (Hoffmann et al., 2022 with the Chinchilla work), these laws establish that language model performance follows predictable power-law curves as a function of three key variables: the number of model parameters, the volume of training data, and the compute budget.

Concretely, these laws show that a model's loss decreases according to a power-law function when any one of these three variables is increased while the other two remain constant. For example, doubling the number of parameters does not double performance, but improves it by a predictable and regular factor. This relationship follows a power law of the form L = a × N^(-α), where L is loss, N is the scaling variable, and α is a characteristic exponent.

The practical impact of scaling laws is immense. They allow research labs to predict model performance before training, simply by extrapolating from smaller models. It is thanks to these laws that investment decisions of several hundred million dollars in compute infrastructure can be made with relative confidence. The Chinchilla work notably showed that many models were 'undertrained' relative to their size, and that a better balance between parameters and data produced superior results at lower cost.

More recently, the community distinguishes 'pre-training scaling laws' from 'inference-time scaling laws' (or test-time compute). The latter, popularized by models like o1 and o3 from OpenAI, show that allocating more computation at generation time—through extended reasoning techniques—can also predictably improve performance, opening a new dimension of optimization.