P Tuning: Definition and Examples

P-Tuning (for Prompt Tuning) is a parameter-efficient fine-tuning method that allows adapting a large language model (LLM) to a specific task without modifying its billions of internal parameters. Instead of rewriting the neural network weights, learnable continuous vectors—called "soft prompts" or "virtual tokens"—are added directly to the input sequence. These vectors are optimized via backpropagation during training.

Unlike classic prompt engineering where instructions are manually written in natural language, P-Tuning works in the model's embedding space. The virtual tokens do not correspond to any real word: they are numerical representations that the optimization algorithm adjusts to maximize performance on the target task. This approach achieves performance comparable to full fine-tuning while modifying only a tiny fraction of the parameters.

There are two main versions of this technique. P-Tuning v1, introduced by Liu et al. in 2021, uses a small LSTM network to generate the embeddings for the learnable prompts. P-Tuning v2, published shortly after, extends the concept by inserting learnable prefixes at every layer of the transformer (not just at the input), significantly improving performance on complex tasks such as text understanding or entity extraction.

P-Tuning belongs to the broader family of PEFT (Parameter-Efficient Fine-Tuning) methods, alongside LoRA and Prefix Tuning. Its main advantage is that it allows the same model instance to serve multiple different tasks: simply load the set of soft prompts corresponding to each task, without duplicating the entire model in memory.