RLHF.

Reinforcement learning from human feedback is a family of training methods that use people’s judgments to shape a model’s behavior. It is often used to improve the usefulness of language-model responses.

In a common approach, people compare candidate responses. Those comparisons train a reward model that estimates which outputs people would prefer. Reinforcement learning then adjusts the language model to receive higher rewards, often with constraints that limit how far it moves from a reference model.

Human feedback is valuable but imperfect. Preferences may be inconsistent, task-specific, or influenced by persuasive writing. A reward model is only a proxy for the qualities people care about, and optimizing it too strongly can exploit its mistakes. Related preference-training methods can skip parts of this pipeline and are not all reinforcement learning in the strict sense.

RLHF does not directly install a complete set of human values or guarantee truthfulness. It optimizes a training signal derived from a particular feedback process and must be evaluated for side effects.