Pruning: Definition and Examples

Pruning is a fundamental optimization technique for artificial intelligence models. Inspired by tree pruning in horticulture, it involves identifying and removing elements of a neural network that contribute least to prediction quality. The goal is to obtain a lighter, faster, and less resource-intensive model without significant performance degradation.

In a deep neural network, many weights have values close to zero or are redundant. Pruning exploits this observation: by removing these less useful connections, the number of model parameters can be significantly reduced. Several strategies exist: unstructured pruning (removing individual weights), structured pruning (removing entire neurons, filters, or layers), and dynamic pruning that adapts during training.

This technique has become crucial with the explosion in size of large language models (LLMs). Models like GPT-4 or LLaMA have billions of parameters, making their deployment costly in memory and computation. Pruning allows creating lightweight versions of these models, facilitating their use on mobile devices, budget-limited servers, or in contexts where latency must be minimal.

In practice, pruning often fits into a broader optimization pipeline that includes quantization and knowledge distillation. A model is first trained normally, then pruned based on an importance criterion (weight magnitude, gradient sensitivity, etc.), and finally briefly retrained (fine-tuned) to recover lost performance. This iterative approach achieves impressive compression rates, sometimes exceeding 90%, with negligible precision loss.