Prefix Tuning: Definition and Examples

Prefix Tuning is a parameter-efficient fine-tuning method introduced by Lisa Li and Percy Liang in 2021. Rather than retraining all billions of parameters of a large language model, this technique adds a small set of continuous vectors — called "prefixes" — to each layer of the transformer. These prefixes are the only elements optimized during training, while all original model weights remain frozen.

Concretely, the prefix acts as a virtual context that steers the model's behavior toward a specific task. Unlike classic fine-tuning, which creates a full copy of the model for each task, Prefix Tuning only requires storing a few thousand additional parameters per task. This typically represents less than 0.1% of the original model's parameters, making the method extremely memory- and storage-efficient.

Prefix Tuning differs from Prompt Tuning (soft prompting) in that the learnable vectors are inserted into all layers of the transformer, not just the input embedding layer. This deep insertion allows the prefix to more finely influence the model's internal representations, generally resulting in better performance, especially on text generation tasks.

This approach is part of a broader movement to democratize the adaptation of large language models. By drastically reducing required resources, Prefix Tuning enables teams with limited means to specialize powerful models for their use cases, while retaining the ability to quickly switch between tasks by simply changing the prefix.