Adapter Tuning: Definition and Examples

Adapter Tuning is a method for adapting language models that relies on inserting small additional neural networks — adapters — between the existing layers of a pre-trained model. During training, only the parameters of these adapters are updated, while the original model's weights remain frozen. This allows adapting a model to a specific task by training only a tiny fraction of its total parameters.

This approach was popularized by the work of Houlsby et al. (2019) at Google Research. The central idea is that a massive model like GPT or BERT already contains a rich representation of language, and that adding a lightweight adaptation layer is sufficient to specialize it. Each adapter typically consists of a projection to a reduced dimension (bottleneck), a non-linear activation function, and then a projection back to the original dimension.

The main advantage of Adapter Tuning is its resource efficiency. Whereas traditional fine-tuning requires storing a complete copy of the model for each task, Adapter Tuning only requires storing the small added modules, typically less than 5% of the model's parameters. This makes it possible to deploy a single base model with multiple interchangeable adapter sets depending on the task.

Adapter Tuning is part of a broader family of techniques called PEFT (Parameter-Efficient Fine-Tuning), which also includes LoRA, Prefix Tuning, and Prompt Tuning. These methods all address the same challenge: how to customize increasingly large models without exploding computational and storage costs.