全课总结 Grand Summary

我们回到一开始提出的那个问题:当你打开 ChatGPT,敲下一句话,点击发送——屏幕背后究竟发生了什么?你看到的、你正在对话的那个东西,到底是什么,它是怎么工作的?走到这里,我们已经有能力把它从头到尾讲清楚了。 Let's loop back to the question we started with: when you open ChatGPT, type a query, and hit go — what is actually happening behind the screen? What is the thing you're seeing, the thing you're talking to, and how does it work? Having come this far, we can now explain it end to end.

首先,你的查询会被切成 token,按对话协议(conversation protocol)的格式插进一个对话对象里;在底层,这一切最终只是一串一维的 token 序列。当你点「发送」,模型就开始往这串序列后面不断追加 token——它的行为就像一个「token 自动补全」:一个一个地续写,直到给出回应。问题随之而来:为什么续写出来的偏偏是这些 token?它们从哪儿来?我们到底在跟什么对话? First, your query is chopped into tokens and inserted into a conversation object following the conversation protocol format; under the hood, all of this is ultimately just a one-dimensional token sequence. When you hit go, the model starts appending tokens to that sequence — it behaves like a 'token autocomplete,' continuing one token at a time until it produces a response. Which raises the question: why exactly these tokens? Where do they come from? What are we actually talking to?

答案是这整门课讲过的三个阶段。第一个阶段是预训练(pretraining):本质上就是从互联网获取知识、把它内化进神经网络的参数里。你可以把它想成:互联网 → 切成 token → 反复做「下一个 token 预测」→ 得到一个 base 模型。网络由此内化了大量来自互联网的知识——但请记住,真正的「人格」还不在这里。 The answer is the three stages this course has covered. The first is pretraining: fundamentally, acquiring knowledge from the internet and internalizing it into the neural network's parameters. Picture it as: internet -> chopped into tokens -> repeated next-token prediction -> a base model. The network thereby internalizes a great deal of knowledge from the internet — but remember, the actual 'personality' isn't here yet.

人格真正出现的地方,是后训练里的第一步:监督微调(SFT)。像 OpenAI 这样的公司会精心策划一个庞大的对话数据集——比如上百万段、横跨各种话题的「人类与助手」对话。尽管这一过程也大量借助合成数据和 LLM 辅助,但它本质上是一项以人为主的数据策划工作:由公司雇来的数据标注员(data labelers),按照他们学习过的标注规范,为各种各样的提问撰写「理想的助手回答」。换句话说,他们是在用例子手把手教神经网络:面对一个提问,该怎么回应。 Personality truly appears in the first step of post-training: supervised fine-tuning (SFT). A company like OpenAI curates a large dataset of conversations — say a million conversations spanning all sorts of topics, each between a human and an assistant. Although the process leans heavily on synthetic data and LLM assistance, it is fundamentally a human-driven data-curation task: data labelers hired by the company, following labeling instructions they've learned, write 'ideal assistant responses' to all kinds of prompts. In other words, they teach the neural network by example how to respond to a prompt.

后训练的另外两步,负责进一步打磨这个模拟器。在可验证的领域(比如数学、代码),我们用强化学习(RL):让模型在大量练习题上反复尝试,自己摸索、发现好的思考策略——那些看起来很像你脑海里内心独白的、解决问题的链路。这一步不是模仿人类标注员,而是在模拟中「涌现」出的思考能力,原则上甚至可能产生连人都没想到过的新解法。而在不可验证的领域(比如创意写作),没有干净的对错可以打分,于是改用 RLHF:先让人类对模型的若干回答做偏好排序,据此训练出一个奖励模型(reward model)来近似人类的喜好,再用它去引导模型。 The other two steps of post-training refine this simulator further. In verifiable domains (e.g. math, code), we use reinforcement learning (RL): the model practices on many problems, trying repeatedly and discovering good thinking strategies on its own — problem-solving chains that look a lot like the internal monologue in your head. This step doesn't imitate a human labeler; it's a thinking ability that 'emerges' in simulation, and in principle could even produce solutions no human has thought of. In unverifiable domains (e.g. creative writing), there's no clean right-or-wrong to score, so we use RLHF instead: have humans rank their preferences among several model answers, train a reward model to approximate human taste from those rankings, then use it to guide the model.

把这条线收束起来:你在 ChatGPT 里对话的,本质上是一个 token 模拟器——先由人类标注员塑形(它学会模仿理想的助手回答),再(在思考型模型里)被强化学习塑形(它练出了自己的思考策略)。不会做 RL 的模型(比如普通的 GPT-4o)给你的,更接近一份「人类标注员的直接模拟」;而思考型模型多走了 RL 这第三步,给你的回应里有了一点真正「新」的、涌现出来的东西。 Tying the line together: what you talk to in ChatGPT is essentially a token simulator — first shaped by human labelers (it learns to imitate ideal assistant responses), then (in thinking models) shaped by reinforcement learning (it has practiced its own thinking strategies). A model that doesn't do RL (e.g. plain GPT-4o) gives you something closer to a 'direct simulation of a human labeler'; a thinking model takes that third RL step, so its responses contain a bit of something genuinely 'new' and emergent.

最后是一点实用的智慧,也是整门课想留给你的态度。这是一个极其令人兴奋的时代:这些模型强大到能显著加速你的工作,很多人每天都在用它们几十上百次。但它们并不完美,带着锋利的边缘——会随机犯傻、随机幻觉、随机跳过该做的心算、随机数不清字母。所以:把它当成工具箱里的一件工具,不要全盘信任。用它找灵感、打初稿、向它提问,但永远去核对、去验证,并对最终产出负责——让「人」留在回路里。做到这一点,它会让你的工作事半功倍。 Finally, a bit of practical wisdom — the attitude this whole course wants to leave you with. This is an extremely exciting time: these models are powerful enough to dramatically accelerate your work, and many people use them dozens or hundreds of times a day. But they are imperfect, with sharp edges — they'll randomly do something dumb, randomly hallucinate, randomly skip the mental arithmetic they should have done, randomly miscount letters. So: treat it as a tool in your toolbox, and don't trust it fully. Use it for inspiration, first drafts, and questions, but always check and verify, and own the final product — keep a human in the loop. Do that, and it will make you far more effective in your work.