Model Distillation: Definition and Examples

Model distillation is a transfer learning technique in which a compact model, called the "student," is trained to imitate the outputs of a larger, more performant model, called the "teacher." Rather than learning directly from raw data, the student trains on the probability distributions produced by the teacher, capturing nuances that classic labels do not contain. The intuition behind this approach rests on the concept of "dark knowledge" introduced by Geoffrey Hinton in 2015. When a large model predicts that a picture of a cat has a 90% chance of being a cat and an 8% chance of being a lynx, this relationship between classes contains rich information that the small model can exploit. By adjusting a temperature parameter during training, the teacher's output distributions are "softened" to make this latent information more accessible. In the context of large language models (LLMs), distillation has become a major strategic issue. Models like GPT-4 or Claude are extremely expensive to deploy. Distillation makes it possible to create lighter, specialized versions for specific tasks that retain a large part of the quality while drastically reducing inference costs and latency. In prompt engineering, distillation takes a practical form: a powerful model is used to generate high-quality examples (synthetic data), then a smaller model is fine-tuned on these examples. This approach democratizes access to high performance and allows deploying effective AI solutions even with budget or infrastructure constraints.