RLHF: Definition and Examples

RLHF, or Reinforcement Learning from Human Feedback, is a training method that fine-tunes large language models after their initial pre-training. Rather than relying solely on raw text data, this technique integrates human judgment directly into the learning process.

The process consists of three main steps. First, the model is pre-trained in a classic manner on vast text corpora. Next, human evaluators compare and rank different responses generated by the model for the same question, creating a dataset of preferences. These preferences are used to train a reward model that learns to predict which response a human would prefer.

Finally, the language model is optimized via reinforcement learning (typically using the PPO algorithm — Proximal Policy Optimization) to maximize the score assigned by this reward model, while staying close to its initial behavior via a KL divergence penalty.

RLHF has played a decisive role in the success of ChatGPT and modern AI assistants. It is this technique that allows models to produce helpful, honest, and harmless responses rather than simply predicting the most likely next word. It remains a very active research area, with variants like DPO (Direct Preference Optimization) simplifying the process.