RLHF:从人类反馈中强化学习 RLHF: Reinforcement Learning from Human Feedback

到目前为止,我们看的所有 RL 问题都属于「可验证」领域:任意一个候选解法,都能拿一个具体答案轻松打分。比如答案是 3,我们就检查模型解法里框出来的答案是不是等于 3;或者用一个「LLM 裁判」——让另一个 LLM 看着解法和标准答案,判断两者是否一致。当下的模型已经足够可靠地做这件事。无论哪种方式,我们都有一个具体答案可对照,而且整个打分是自动的、不需要任何人参与。 So far, every RL problem we've looked at is in a 'verifiable' domain: any candidate solution can be scored easily against a concrete answer. If the answer is 3, we check whether the boxed answer in the model's solution equals 3; or we use an 'LLM judge' — another LLM that looks at the solution and the reference answer and decides whether they're consistent. Today's models are reliable enough to do this. Either way, we have a concrete answer to check against, and the whole scoring is automatic, with no human in the loop.

问题在于,这套策略没法用在「不可验证」领域。典型的就是各种创意写作任务:写一个关于鹈鹕的笑话、写一首诗、给一段话写摘要。这些任务里,给不同的解法打分变得困难得多。比如写鹈鹕笑话——我们当然可以生成一大堆不同的笑话,但难点在于:怎么给它们打分?哪个更好笑,本身就没有客观的、可机器检查的标准答案。 The problem is that this strategy doesn't apply in 'unverifiable' domains. The classic ones are creative-writing tasks: write a joke about pelicans, write a poem, summarize a paragraph. In these tasks, scoring different solutions becomes much harder. Take pelican jokes — we can certainly generate a pile of different jokes, but the difficulty is: how do we score them? Which one is funnier has no objective, machine-checkable concrete answer.

原则上我们当然可以请人来看这些笑话、逐个打分。但放进 RL 的循环里就崩了:做强化学习要进行成千上万次更新,每次更新要看上千个 prompt,每个 prompt 又要看上百乃至上千个生成。粗略一算,这可能意味着要请人评判上亿次——上亿个(而且多半很烂的)笑话。这是个完全无法扩展的策略,我们需要某种自动的办法。 In principle we could of course have humans look at the jokes and score them one by one. But drop that into the RL loop and it collapses: reinforcement learning runs many thousands of updates, each update looks at thousands of prompts, and each prompt has hundreds or thousands of generations. Roughly multiplied out, that can mean asking humans to judge on the order of a billion times — a billion (mostly terrible) jokes. That's a completely unscalable strategy; we need some automatic approach.

解法来自一篇论文(Christiano 等人,出自当时的 OpenAI;其中不少人如今是 Anthropic 的联合创始人),提出了「从人类反馈中强化学习」(RLHF)。核心招数是「间接」:我们只让人类介入一点点,然后训练一个完全独立的神经网络,叫做「奖励模型」(reward model),让它去模仿人类的打分。一旦这个网络变成了「人类偏好的模拟器」,我们就可以转而对着它跑 RL——不再问真人,而是问这个模拟出来的人。模拟器不会是完美的人,但只要它在统计上和人类判断足够接近,RL 就能奏效;实践中也确实如此。 The solution comes from a paper (Christiano et al., out of what was then OpenAI; many of those people are now co-founders of Anthropic) that proposed 'reinforcement learning from human feedback' (RLHF). The core trick is indirection: we involve humans only a little, then train a completely separate neural network called a 'reward model' to imitate human scores. Once that network becomes a 'simulator of human preferences,' we run RL against it instead — no longer asking a real human, but asking the simulated one. The simulator won't be a perfect human, but as long as it's statistically close enough to human judgment, RL works; and in practice it does.

训练奖励模型大致是这样:拿一个 prompt(比如「写一个关于鹈鹕的笑话」),生成五个不同的笑话。我们不让人直接打绝对分,而是请人把这五个笑话从最好到最差排个序——因为对人来说,给出一个排序,比给出精确分数要容易。这个排序,就是人类对训练过程的全部贡献。另一边,奖励模型(一个独立的神经网络,通常也是 Transformer,但它不生成语言,只输出分数)会读入「prompt + 一个候选笑话」,吐出一个介于 0 到 1 之间的分数(0 最差、1 最好)。 Training the reward model goes roughly like this: take a prompt (say 'write a joke about pelicans') and generate five different jokes. Instead of asking a human for absolute scores, we ask them to ORDER the five jokes from best to worst — because for a human, giving an ordering is easier than giving precise scores. That ordering is the human's entire contribution to training. On the other side, the reward model (a separate neural network, usually also a Transformer, but it doesn't generate language — it only outputs a score) reads in 'prompt + one candidate joke' and emits a single number between 0 and 1 (0 worst, 1 best).

然后我们把奖励模型给出的分数,和人类给出的排序做对比,用一个损失函数衡量两者的「一致程度」,并据此更新奖励模型。直觉上:如果人觉得某个笑话最好笑、模型也给了高分(比如 0.8),那方向对了,只是分数还该再高一点——更新后也许变成 0.81;如果人觉得某个排第二、模型却只给了 0.1,这是严重分歧,更新后这个分数就该大幅上调;如果人觉得某个最差、模型却给了高分,更新后这个分数就该下来。如此反复在人类排序数据上更新,奖励模型就越来越像一个「人类偏好模拟器」,我们就能对着它跑 RL。 Then we compare the reward model's scores against the human's ordering, use a loss function to measure how 'consistent' the two are, and update the reward model accordingly. Intuitively: if the human thought a joke was funniest and the model also gave it a high score (say 0.8), the direction is right but the score should be even higher — after an update maybe 0.81; if the human ranked one second but the model gave only 0.1, that's a big disagreement, and after an update that score should rise sharply; if the human thought one was worst but the model gave it a high score, after an update that score should come down. Repeating this over human ordering data, the reward model becomes more and more like a 'simulator of human preferences,' and we can run RL against it.

先说 RLHF 的好处。第一,它让我们得以把 RL 这套极其强大的技术,用到任意领域,包括那些不可验证的领域:摘要、写诗、写笑话,以及数学和代码之外的各种创意写作。第二,经验上,正确地做 RLHF 确实能让模型变好一点——为什么会这样并没有特别确凿的定论,但最可能的解释还是上面说的「判别比生成容易」:我们不再要求人去做极难的生成(创意写作)任务,只要求他们分辨好坏、挑出最好的;系统随后自己去发现那些会被人类评为「好」的回答。这一步间接,反而带来了更高质量的监督信号。 First, the upside of RLHF. One: it lets us apply RL — an extremely powerful set of techniques — to arbitrary domains, including unverifiable ones: summarization, poetry, jokes, and all kinds of creative writing beyond math and code. Two: empirically, doing RLHF correctly does make the model a bit better — why isn't firmly established, but the likeliest explanation is the discriminator–generator gap above: we no longer ask humans to do the extremely hard generation (creative-writing) task, only to discriminate good from bad and pick the best; the system then discovers for itself the responses that humans would grade well. That step of indirection yields higher-quality supervision.

但 RLHF 也带来重大的坏处。我们做 RL 时,对照的不是真正的人类判断,而是一个有损的人类模拟。更要命的是一个微妙又「狡猾」的现象:RL 极其擅长找到「骗过模拟器」的办法。奖励模型是个有上十亿参数的庞大 Transformer,它在「模仿」人类,但终究只是模拟。于是存在一些根本不在它训练集里的输入,这些输入会莫名其妙地拿到虚高的分数。比如你把 RLHF 跑上一千步,前几百步笑话确实在变好,但之后会「跌下悬崖」:你会发现排名第一的「鹈鹕笑话」竟然是诸如「the the the」这种毫无意义的东西——可奖励模型偏偏给它打了接近 1 的满分。 But RLHF also brings major downsides. When we run RL, we're optimizing against not actual human judgment but a lossy simulation of humans. Worse is a subtle, 'devious' phenomenon: RL is extremely good at finding ways to fool the simulator. The reward model is a massive Transformer with billions of parameters; it 'imitates' humans, but it's still only a simulation. So there exist inputs that were never in its training set yet inexplicably earn spuriously high scores. For example, run RLHF for a thousand steps: in the first few hundred the jokes do improve, but then they 'fall off a cliff' — you find that the top-ranked 'pelican joke' is something nonsensical like 'the the the,' yet the reward model gives it a near-perfect score close to 1.

这类输入叫「对抗性样本」(adversarial examples)——它们钻进了模型输入空间的缝隙里,在顶端给出荒诞结果。你也许会想:那就把「the the the」加进训练集、给它极低分不就行了?确实,模型会学会给这一条打低分;但问题是,这种无意义的对抗性样本几乎是无穷多的。你不断把新发现的荒诞输入加回去、给低分,也永远赢不了这场游戏:只要 RL 跑得足够久,它总能再找到一个新的办法去钻奖励模型的空子,用毫无意义的结果拿到虚高分。根本原因是:打分函数本身是个巨大的神经网络,而 RL 极其擅长找到骗过它的路径。 Such inputs are called 'adversarial examples' — they slip into the nooks and crannies of the model's input space and produce nonsense at the top. You might think: just add 'the the the' to the training set with a terrible score? Indeed the model will learn to score that one low; but the problem is that such nonsensical adversarial examples are essentially infinite. You keep adding newly discovered nonsense back in with low scores, and you still never win the game: run RL long enough and it always finds a fresh way to game the reward model, earning spuriously high scores with meaningless output. The root cause: the scoring function is itself a giant neural network, and RL is extremely good at finding ways to trick it.