Quantization: Definition and Examples

Quantization is a process that converts the parameters of a language model, typically stored as high-precision floating point numbers (FP32 or FP16), into lower-precision numerical representations like INT8 or INT4. This precision reduction drastically decreases the model's memory footprint and speeds up inference computations.

Concretely, a model like LLaMA 70B requires about 140 GB of memory in FP16. By using 4-bit quantization (Q4), the same model can fit into roughly 35 GB, making it executable on consumer hardware. This compression comes with a slight loss in quality, but modern techniques such as GPTQ, AWQ, or GGUF minimize this degradation remarkably.

There are two main approaches: post-training quantization (PTQ), applied to a trained model, and quantization-aware training (QAT), which incorporates the reduced precision constraint directly during training. PTQ is more widespread because it does not require retraining the model, while QAT generally offers better results at the cost of higher computational expense.

For prompt users, quantization is important because it determines the quality of responses when using a local model. A Q8 quantized model is nearly identical to the original, while Q2 shows noticeable degradation, especially on complex reasoning tasks or code generation. Choosing the right quantization level is a trade-off between available resources and expected quality.