LoRA: Definition and Examples

LoRA, short for Low-Rank Adaptation, is a method for adapting artificial intelligence models developed by Microsoft researchers in 2021. Instead of retraining all billions of parameters of a model, LoRA injects small low-rank matrices into the neural network layers. This achieves results comparable to full fine-tuning while drastically reducing required resources (GPU memory, computation time, storage).

The principle is based on a mathematical observation: during fine-tuning, the modifications to model weights actually occupy a low-dimensional space. LoRA exploits this property by decomposing weight updates into two small matrices (A and B) whose product approximates the full update. Concretely, instead of modifying a weight matrix of size 10,000 × 10,000 (100 million parameters), two matrices of size 10,000 × 8 and 8 × 10,000 (160,000 parameters) are used, a reduction of over 99%.

In image generation, LoRA has become extremely popular with Stable Diffusion. Users create and share LoRAs trained on artistic styles, characters, or specific concepts. A LoRA file typically weighs only a few dozen megabytes, compared to several gigabytes for a full model, facilitating sharing and combination of multiple adaptations.

For language models (LLMs), LoRA enables companies and researchers to adapt models like LLaMA or Mistral to specific business domains (legal, medical, technical) without requiring expensive GPU servers. This democratization of fine-tuning has significantly accelerated the adoption of generative AI in various professional contexts.