Chinchilla Optimal: Definition and Examples

The concept of Chinchilla Optimal comes from a 2022 research paper by the DeepMind team, titled 'Training Compute-Optimal Large Language Models'. This study challenged the prevailing approach of scaling model size (number of parameters) without proportionally increasing training data. The Chinchilla model, which gives its name to the principle, demonstrated that a smaller model trained on more data could outperform a larger model trained on less data, given the same compute budget.

The empirical rule derived from this research states that for every doubling of model parameters, the amount of training tokens should also double. Concretely, a 70 billion parameter model should be trained on approximately 1.4 trillion tokens to be 'Chinchilla optimal'. Before this discovery, models like GPT-3 (175 billion parameters) were trained on only 300 billion tokens, making them significantly under-trained by this criterion.

This principle has profoundly transformed the training strategy of language models in the industry. Rather than simply stacking parameters, research labs began to invest heavily in collecting and curating high-quality training data. Models like Meta's Llama 2 were explicitly designed to exceed the Chinchilla optimal ratio, favoring over-training on more data to achieve better inference performance.

It is important to note that the Chinchilla optimal ratio applies strictly to optimizing the training compute budget. In practice, many labs deliberately choose to over-train their models (i.e., use more data than the ratio recommends) because a smaller but better-trained model is cheaper to deploy and run in production, even if its initial training was longer.