LLM 全栈学习

深入理解大语言模型

基于 Andrej Karpathy《Deep Dive into LLMs》—— 从预训练到 RLHF 的交互式中文课程。

0%

已读 0 / 22 章

课程章节

  1. 01 引言:把 ChatGPT 拆开看Introduction: Taking ChatGPT Apart 建立思考大模型的心智模型——它擅长什么、不擅长什么、有哪些需要警惕的边角。Build a mental model for what LLMs are: where they shine, where they fail, and the sharp edges to watch for.
  2. 02 预训练数据:把互联网喂给模型Pretraining Data: Feeding the Internet to the Model 预训练的第一步是「下载并处理整个互联网」。我们看看从 Common Crawl 的原始网页,经过层层过滤,最终变成像 FineWeb 那样约 44TB、约 15 万亿 token 的高质量语料的全过程。The first step of pretraining is to download and process the internet. We trace the journey from Common Crawl's raw web pages, through layers of filtering, into a high-quality corpus like FineWeb — roughly 44TB and about 15 trillion tokens.
  3. 03 分词:文本如何变成 tokenTokenization: How Text Becomes Tokens 神经网络只接受「一维的、有限种类的符号序列」。我们看看为什么不能直接喂比特,如何从字节出发,再用字节对编码(BPE)在序列长度和词表大小之间做权衡,最终得到 GPT-4 那约 10 万个 token 的词表。Neural networks only accept a one-dimensional sequence drawn from a finite set of symbols. We look at why we can't feed raw bits, how we start from bytes, and how byte pair encoding (BPE) trades off sequence length against vocabulary size — ending up at GPT-4's vocabulary of roughly 100,000 tokens.
  4. 04 神经网络的输入与输出Neural Network Inputs and Outputs 把语料分词后,我们就得到约 15 万亿个 token。训练神经网络的核心,是建模「token 如何一个接一个地出现」。我们先不打开网络这个黑盒,只看清它的输入(变长的 token 窗口)和输出(覆盖整个词表的概率分布),以及训练时如何一步步调整它。Once the corpus is tokenized we have about 15 trillion tokens. Training the network is about modeling how tokens follow one another. Without opening the black box yet, we look at its input (a variable-length window of tokens) and its output (a probability distribution over the whole vocabulary), and how training nudges it step by step.
  5. 05 神经网络内部:TransformerInside the Neural Network: The Transformer 上一章我们只看了网络的输入和输出,把它当成黑盒。现在打开盒子看看里面:输入的 token 与数十亿个参数(权重)被搅进一个巨大的数学表达式;参数最初是随机的,训练就是慢慢调整它们。我们会认识 Transformer 这种架构,理解里面的「神经元」其实极其简单、没有记忆,以及为什么真正重要的只是「一个从输入到输出的固定函数」。Last chapter we treated the network as a black box, looking only at its inputs and outputs. Now we open the box: the input tokens get mixed with billions of parameters (weights) inside one giant mathematical expression; the parameters start random, and training slowly adjusts them. We meet the Transformer architecture, see that its 'neurons' are extremely simple and memoryless, and understand why what really matters is just 'a fixed function from input to output.'
  6. 06 推理:从模型生成文本Inference: Generating Text from the Model 训练之外的另一个关键阶段是推理(inference)——从模型里生成新数据。做法很简单:给一段前缀 token,网络吐出下一个 token 的概率分布,我们「掷一枚有偏的硬币」采样出一个 token,接到末尾,再重复。因为是采样,同样的前缀会得到不同的输出;生成的文本是对训练数据的「remix」,统计上相似但并非逐字照搬。你在 ChatGPT 上对话时,发生的全部就是推理。Beyond training, the other key stage is inference — generating new data from the model. It's simple: give it a prefix of tokens, the network outputs a probability distribution over the next token, we 'flip a biased coin' to sample one token, append it, and repeat. Because it's sampling, the same prefix yields different outputs; the generated text is a 'remix' of the training data — statistically similar but not verbatim. When you chat with ChatGPT, all that's happening is inference.
  7. 07 实战:训练 GPT-2Hands-On: Training GPT-2 用一个具体例子把训练和推理串起来:OpenAI 在 2019 年发布的 GPT-2。它是第一次「可辨认的现代技术栈」凑齐——16 亿参数、上下文长度 1024、约 1000 亿 token,按今天的标准都很小。我们会看训练时研究者盯着的那个数字——损失(loss),理解为什么训练成本从 2019 年的约 4 万美元降到今天的约一两百到六百美元,以及为什么 GPU 成了这场「淘金热」。We tie training and inference together with a concrete example: OpenAI's GPT-2, released in 2019. It's the first time a recognizably modern stack came together — 1.6B parameters, a context length of 1,024, and about 100B tokens, all small by today's standards. We look at the number researchers stare at during training — the loss — understand why training cost fell from about $40k in 2019 to roughly $100–600 today, and why GPUs became the gold rush.
  8. 08 Base 模型:互联网文本模拟器Base Models: Internet Text Simulators 预训练跑完后得到的东西叫 base 模型——一个 token 级的互联网文档模拟器,而不是助手。我们看看「发布一个模型」到底意味着什么(代码 + 参数),以 Meta 的 Llama 3.1 405B base 为例,玩一玩它的几种典型行为:逐字背诵(regurgitation)、自信地编造(hallucination)、从提示里的例子现学规律(in-context / few-shot learning),甚至用一段巧妙的对话提示把 base 模型「调教」成助手。最后理解:参数是对互联网的一种有损压缩。What you get after pretraining is a base model — a token-level internet-document simulator, not an assistant. We look at what 'releasing a model' really means (code + parameters), use Meta's Llama 3.1 405B base as the example, and play with its characteristic behaviors: verbatim regurgitation, confident hallucination, in-context / few-shot learning from examples in the prompt, and even coaxing a base model into acting like an assistant with a clever conversation prompt. Finally we see the parameters as a lossy compression of the internet.
  9. 09 从预训练到后训练From Pretraining to Post-Training 一个承上启下的短章。我们先回顾:预训练把互联网文档拆成 token、用神经网络预测序列,产出一个 base 模型——一个 token 级的互联网文档模拟器。但我们要的是助手:能问、能答。于是把 base 模型交给第二阶段——后训练(post-training)。后训练在计算上比预训练便宜、短得多(几小时 vs 约三个月),用的训练算法相同,只是把数据集从互联网文档换成对话。后续章节会展开对话数据、SFT 和 RLHF。A short bridging chapter. First we recap: pretraining breaks internet documents into tokens and uses a neural network to predict sequences, yielding a base model — a token-level internet-document simulator. But what we want is an assistant: one we can ask and that answers. So we hand the base model to a second stage — post-training. Post-training is computationally far cheaper and shorter than pretraining (hours vs about three months), uses the same training algorithm, and just swaps the dataset from internet documents to conversations. Later chapters unpack conversation data, SFT, and RLHF.
  10. 10 后训练数据:对话Post-Training Data: Conversations 后训练的核心,是把训练数据从「互联网文档」换成「对话」——人类与助手之间的多轮问答。我们没法像写代码那样显式地给助手编程,只能通过对话数据集「以例编程」:用例子隐式地教会它如何回应。这些对话会被一套协议(像 <|im_start|>、角色、内容、<|im_end|> 这样的特殊 token)编码成一维 token 序列,这些特殊 token 是后训练才新引入的。我们追溯到 OpenAI 2022 年的 InstructGPT,看人类标注员如何按「有用、真实、无害」的标注规范写出理想回答;再看现代数据集(如 UltraChat)如何大量借助 LLM 合成。最终的直觉:和 ChatGPT 对话,约等于在问「一个受过良好指导的人类标注员会怎么回答」。The core of post-training is swapping the training data from 'internet documents' to 'conversations' — multi-turn Q&A between a human and an assistant. We can't program the assistant explicitly like writing code; we can only program it implicitly through a dataset of conversations, teaching it how to respond by example. These conversations are encoded into a one-dimensional token sequence by a protocol (special tokens like <|im_start|>, a role, the content, and <|im_end|>), and those special tokens are newly introduced in post-training. We trace this back to OpenAI's 2022 InstructGPT, seeing how human labelers write ideal responses following 'helpful, truthful, harmless' labeling instructions; then how modern datasets (e.g. UltraChat) are largely LLM-synthesized. The takeaway intuition: talking to ChatGPT ≈ asking 'what would a well-instructed human labeler say.'
  11. 11 幻觉、工具与工作记忆Hallucinations, Tools, and Working Memory 本章谈「LLM 心理学」的第一组现象。幻觉(hallucination)就是模型自信地编造信息——根源在于训练集里「谁是某某」这类问题全都被自信地正确回答,模型于是学会了无论知不知道都用同样自信的口吻作答。我们看两种缓解办法:其一,探测模型、找出它「不知道」的边界,在训练集里补上「我不知道」的样例,让它学会把内部的不确定感和「拒答」关联起来;其二,给模型工具(如网页搜索),它发出特殊 token 调用工具,结果被粘进上下文窗口,再据此作答。由此引出核心区分:上下文窗口是模型可直接访问的「工作记忆」,而参数里的知识只是对训练的「模糊回忆」——就像几个月前读过的东西 vs 此刻摆在你眼前的东西。This chapter covers the first cluster of 'LLM psychology' phenomena. Hallucination is when a model confidently makes things up — rooted in the fact that 'who is X' questions in the training set are all answered confidently and correctly, so the model learns to answer in that same confident tone whether or not it actually knows. We look at two mitigations: first, probe the model to find the boundary of what it doesn't know, and add 'I don't know' examples to the training set so it learns to associate internal uncertainty with refusal; second, give the model tools (e.g. web search) — it emits special tokens to call a tool, the result is pasted into the context window, and it answers from that. This leads to the core distinction: the context window is the model's directly-accessible 'working memory,' whereas knowledge in the parameters is just a 'vague recollection' of training — like something you read months ago vs. something in front of you right now.
  12. 12 模型的自我认知The Model's Sense of Self 一个简短的概念章。网上常有人问 LLM「你是什么模型、谁造的你」,但这个问题本身有点没有意义:模型没有持久的自我——每次对话都从零启动、处理 token、然后清空。默认情况下,它对「你是谁」给出的只是统计上的最佳猜测;因为互联网上满是 ChatGPT/OpenAI 的内容,很多模型就顺口说自己是 OpenAI 造的 ChatGPT——这只是「幻觉出来的标签」,不是真实身份。开发者可以「编程」一个人设:要么在 SFT 数据里硬编码一批关于自我的对话,要么在每段对话开头注入一条系统消息(隐藏在上下文里的不可见 token)。所以模型的「自我」不过是又一批被编程进去的 token,而非真正的自我认知。A short conceptual chapter. People online often ask an LLM 'what model are you, who built you,' but the question is itself a bit nonsensical: the model has no persistent self — each conversation boots up from scratch, processes tokens, then clears. By default, its answer to 'who are you' is just a statistical best guess; because the internet is full of ChatGPT/OpenAI content, many models offhandedly say they're ChatGPT built by OpenAI — a 'hallucinated label,' not a real identity. A developer can 'program' a persona: either hardcode a batch of self-referential conversations in the SFT data, or inject a system message at the start of each conversation (invisible tokens hidden in the context). So a model's 'self' is just more programmed tokens, not genuine self-knowledge.
  13. 13 模型需要 token 来思考Models Need Tokens to Think 每个 token 只能享有有限的计算量(有限的网络层数),所以模型没法在单个 token 里做无限推理。结论:把推理「摊开」到许多 token 上(把过程一步步写出来),比直接蹦出答案要好得多。我们用「Emily 买苹果」的数学题做对比:一上来就报答案,等于逼模型在一个 token 里塞下全部计算——很可能算错;而先列中间步骤、慢慢逼近答案,是好得多的训练标签。推论:遇到困难算术,与其让模型「心算」,不如让它调用工具(代码解释器),因为它本就不擅长在脑内可靠计算。token 序列就是模型用来做计算的「草稿纸 / 工作记忆」。Each token gets only a finite amount of computation (a finite number of network layers), so the model can't do unbounded reasoning within a single token. Conclusion: spreading reasoning across many tokens (writing the work out step by step) is far better than blurting the answer. We compare with the 'Emily buys apples' math problem: stating the answer up front forces the model to cram all the computation into one token — likely getting it wrong; laying out intermediate steps and approaching the answer slowly is a far better training label. Corollary: for hard arithmetic, rather than having the model do 'mental math,' have it call a tool (a code interpreter), since it isn't good at reliable in-head computation. The token sequence is the model's 'scratchpad / working memory for computation.'
  14. 14 参差不齐的智能Jagged Intelligence 「参差不齐的智能(jagged intelligence)」指的是:模型可以在某些任务上超越人类,却在小孩都觉得简单的事情上翻车。经典例子:「9.11 比 9.9 大吗?」(经常答错)、「strawberry 里有几个 r」(数不清字母)。根源有几条:分词(模型看到的是 token 块,不是单个字符)、每个 token 的计算量有限、训练数据里的种种怪癖(比如 9.11 让模型联想到圣经章节号)。实践启示:别假设模型在所有任务上能力均匀;在这些「锋利边缘」附近多加验证,该用工具时就用工具。'Jagged intelligence' refers to the fact that models can be superhuman at some tasks yet fail at things a child finds trivial. Classic examples: 'is 9.11 bigger than 9.9?' (often wrong) and 'how many r's in strawberry' (can't count the letters). The roots are several: tokenization (the model sees chunks of tokens, not individual characters), finite per-token computation, and quirks in the training data (e.g. 9.11 reminds the model of Bible verse numbers). Practical takeaway: don't assume uniform competence; verify extra carefully near these 'sharp edges,' and use tools where appropriate.
  15. 15 从监督微调到强化学习From Supervised Fine-Tuning to Reinforcement Learning 训练 LLM 和送孩子上学很像。预训练 = 读教材的说明性文字(背景知识);监督微调(SFT) = 看专家给出的「例题详解」,照着模仿;但还有第三阶段——做「练习题」,这就是强化学习(RL)。本章讲清楚 SFT 的根本局限:用「Emily 买苹果」这道题,四个候选解法都能算出每个苹果 3 美元,但作为人类标注员,你并不知道哪条解法对「模型」最好。一个解法有两重目的——(1)算对答案,(2)讲得让人看得舒服;而对人友好的呈现,未必是模型用最少计算量、最可靠地到达答案的 token 序列(回忆:每个 token 计算量有限)。人和模型的「认知」不同:你觉得轻松的一步,模型可能跨不过去;你写下的某个跳跃,模型参数里可能根本没有。结论:与其替模型猜「理想解法」,不如让它自己练——尝试很多解法,留下能得到正确答案的那些。这正是下一章强化学习要做的事。Training an LLM is a lot like sending a child to school. Pretraining = reading the expository text of a textbook (background knowledge); supervised fine-tuning (SFT) = studying the expert's 'worked solutions' and imitating them; but there is a third stage — doing the 'practice problems,' which is reinforcement learning (RL). This chapter pins down the fundamental limit of SFT: with the 'Emily buys apples' problem, four candidate solutions all reach $3 per apple, yet as a human labeler you don't actually know which solution is best for the MODEL. A solution serves two purposes — (1) reach the right answer, (2) read nicely for a human; but the human-friendly presentation may not be the token sequence by which the model reaches the answer most reliably and with the least compute (recall: finite compute per token). Human and model 'cognition' differ: a step that feels trivial to you might be too big a leap for the model; a jump you write down may not exist in the model's parameters. Conclusion: rather than guessing the 'ideal solution' for the model, let it practice — try many solutions and keep the ones that reach the correct answer. That is exactly what the next chapter's reinforcement learning does.
  16. 16 强化学习Reinforcement Learning 强化学习的核心其实很简单:猜了再验(guess and check)。拿一个 prompt,让模型并行生成大量候选解法(实践中可能上千甚至上百万个);检查哪些到达了正确的最终答案;然后训练模型「多做那些成功解法里做过的事、少做失败解法里做过的事」。关键在于:奖励是自动的——最终答案对不对就是信号,不需要人来评判过程。所以它在「可验证」的领域(数学、代码)效果极好。这些训练序列不来自人类专家,而来自模型自己;模型在「游乐场」里反复练习,自己发现哪些 token 序列可靠地通向答案——这些序列不做莫名跳步、充分利用了模型自身的知识。这就是「推理 / 思考」模型的由来。RL 比 SFT 新、不那么标准化,各家实验室对细节守口如瓶;DeepSeek 是少数公开谈论它的(下一章)。The core of reinforcement learning is actually simple: guess and check. Take a prompt, have the model generate many candidate solutions in parallel (in practice possibly thousands or even millions); check which reach the correct final answer; then train the model to 'do more of what the winning solutions did, and less of what the losing ones did.' The key: the reward is automatic — whether the final answer is correct is the signal, with no human needed to judge the process. So it works extremely well in 'verifiable' domains (math, code). These training sequences don't come from human experts but from the model itself; the model practices in a 'playground,' discovering for itself which token sequences reliably lead to the answer — sequences that make no baffling leaps and fully use the model's own knowledge. This is the origin of 'reasoning / thinking' models. RL is newer and less standardized than SFT, and labs keep the details private; DeepSeek is one of the few to discuss it openly (next chapter).
  17. 17 DeepSeek-R1:推理的涌现DeepSeek-R1: The Emergence of Reasoning DeepSeek-R1 论文首次公开展示了把强化学习用在推理问题上的全过程。最惊人的发现不是准确率上升(这在意料之中),而是质变:在 RL 训练过程中,模型的回答会「自发地越来越长」。它自己学会了推理——从不同角度重新推导、复核自己的工作、回溯纠错(也就是「思维链」)。这些「认知策略」不是被显式教出来的,而是从「只追求答对」的优化里涌现出来的;它很像人类解难题时打的草稿。这就是「思考模型」(相对于早期只会模仿的 SFT 模型)。你能在 DeepSeek-R1、OpenAI 的 o 系列这样的模型里直接看到这种思考,数学/推理基准上的准确率随之大幅跃升。这些推理模型现已可用(如 chat.deepseek.com、together.ai,以及 OpenAI 的 o 系列),但「思考」会在推理时消耗更多算力、也更慢。The DeepSeek-R1 paper was the first to publicly show the full process of applying reinforcement learning to reasoning problems. The most striking finding isn't that accuracy rises (that's expected) but a qualitative change: during RL training, the model's responses 'spontaneously grow longer.' It learns to reason on its own — re-deriving things from different perspectives, double-checking its own work, backtracking to fix mistakes (i.e. 'chains of thought'). These 'cognitive strategies' aren't taught explicitly; they emerge from optimization that only rewards getting the answer right, much like the scratch work a human does on a hard problem. This is a 'thinking model' (versus the earlier SFT models that merely imitated). You can see this thinking directly in models like DeepSeek-R1 and OpenAI's o-series, and accuracy on math/reasoning benchmarks jumps sharply. These reasoning models are now available (e.g. chat.deepseek.com, together.ai, and OpenAI's o-series), but 'thinking' costs more compute at inference and is slower.
  18. 18 AlphaGo 的启示Lessons from AlphaGo 「强化学习极其强大」并不是 LLM 时代才有的新发现。一个经典例证是围棋:DeepMind 的 AlphaGo。它有两种训练方式——监督学习(模仿人类高手的棋局)和强化学习(自我对弈,奖励=赢棋)。结果是 RL 版本超越了监督版本,并击败了世界冠军。监督学习因为只是模仿人类,会「封顶」在人类水平、永远无法真正超越顶尖棋手;而 RL 不受人类表现的约束,可以发现人类未知的新策略。最著名的就是「第 37 手」——一步人类几乎不会下(被估计为万分之一概率)、事后却被证明绝妙的棋。它对应到 LLM 上的启示是:SFT 模仿人类(以人为上限),RL 则可能发现超越模仿的新推理策略。于是有一个开放问题:LLM 版本的「第 37 手」会是什么?也许是人类想不到的类比、全新的思考策略,甚至模型自创的、不再是英语的「思考语言」。前提是要有海量、多样的可练习题目。一个关键告诫:围棋有清晰的胜负奖励,而开放式语言任务很难定义奖励——这为下一章的 RLHF 埋下伏笔。'Reinforcement learning is extremely powerful' is not a discovery unique to the LLM era. A classic example is the game of Go: DeepMind's AlphaGo. It was trained two ways — supervised learning (imitating human expert games) and reinforcement learning (self-play, reward = winning). The RL version surpassed the supervised one and beat the world champion. Supervised learning, being mere imitation of humans, 'caps out' at human level and can never truly exceed the top players; RL is not constrained by human performance and can discover new strategies unknown to humans. The most famous is 'Move 37' — a move no human would play (estimated at about 1 in 10,000 probability) that proved brilliant in hindsight. The lesson for LLMs: SFT imitates humans (capped at the human ceiling), while RL can discover new reasoning strategies beyond imitation. So there's an open question: what is the LLM equivalent of 'Move 37'? Perhaps analogies humans couldn't make, an entirely new thinking strategy, even a model-invented 'thinking language' that isn't English. The prerequisite: a huge, diverse set of problems to practice on. A key caveat: Go has a crisp win/lose reward, while open-ended language tasks are hard to reward — foreshadowing the next chapter's RLHF.
  19. 19 RLHF:从人类反馈中强化学习RLHF: Reinforcement Learning from Human Feedback 前面讲的 RL 都活在「可验证」领域(数学、代码):任何候选解法都能拿一个具体答案自动打分——检查框里的答案,或用 LLM 当裁判,全程无需人。可惜很多领域是「不可验证」的:写个笑话、写首诗、写段摘要……没有标准答案,难以自动评分。最朴素的想法是让人给每个生成打分,但 RL 要做上千次更新 × 上千个 prompt × 每个 prompt 上千个生成,这是上亿次人工评判,根本不可行。RLHF(Christiano 等人 / OpenAI)的解法是「间接」:只让人对一小批样本做排序(排序比打绝对分容易),用这些排序训练一个独立的「奖励模型」(一个神经网络)去模仿人类偏好,再用这个奖励模型自动给海量生成打分,从而在不可验证领域里跑 RL。好处:让我们能在不可验证领域跑 RL,甚至还能轻微改善可验证领域。但有个大坑:奖励模型只是人类判断的「模拟」;如果 RL 跑太久,模型会找到一些荒诞的对抗性输入,骗到虚高分数(reward hacking / 钻奖励模型空子)。所以 RLHF 不能像可验证 RL(那种 AlphaGo 式的「魔法」RL)那样无限跑——它更像一次「微调」,跑几百步、有改善就收手。The RL we've covered so far lives in 'verifiable' domains (math, code): any candidate solution can be auto-scored against a concrete answer — check the boxed answer, or use an LLM judge — with no human in the loop. Unfortunately many domains are 'unverifiable': write a joke, write a poem, summarize a paragraph ... there's no concrete answer, so they're hard to auto-score. The naive idea is to have humans score every generation, but RL does thousands of updates × thousands of prompts × thousands of generations each — hundreds of millions of human judgments, which is unworkable. RLHF (Christiano et al. / OpenAI) solves this with indirection: have humans only RANK a small set of samples (ranking is easier than absolute scoring), use those rankings to train a separate 'reward model' (a neural network) that imitates human preferences, then use that reward model to auto-score the massive number of generations — letting us run RL in unverifiable domains. Upside: it lets us run RL in unverifiable domains and even slightly improves verifiable ones. But there's a big caveat: the reward model is only a 'simulation' of human judgment; if you run RL too long, the model finds absurd adversarial inputs that earn spuriously high scores (reward hacking / gaming the reward model). So RLHF can't be run indefinitely like verifiable-domain RL (the AlphaGo-style 'magic' RL) — it's more of a fine-tune: run a few hundred steps, take the improvement, and stop.
  20. 20 未来展望What's Coming Next 课程接近尾声,这一章瞥一眼「正在到来」的能力。第一,模型会迅速多模态化:不只处理文本,还能原生地听、说(音频)和看、画(图像)。原理上其实没变——把音频(频谱切片)和图像(图块 patch)也切成 token,塞进上下文窗口,用同一套方法训练。第二,智能体(agents):今天我们还是把单个任务「端到盘子上」交给模型;未来模型会作为 agent,连贯地、能纠错地执行长时间、多步骤的任务——耗时几十秒、几分钟乃至几小时,由人来监督(类比工厂里的「人机比」,数字世界会有「人–智能体比」)。第三,模型会越来越普遍、隐形地嵌入到工具里(比如你的电脑 / IDE),并开始能代替你执行操作(键盘鼠标)。第四,还有大量研究待做,比如「测试时学习 / 思考」:现在模型部署后参数固定,唯一的「学习」是上下文窗口里的 in-context learning;而上下文是有限且宝贵的资源,面对多模态、长时间任务会迅速膨胀,光靠「把上下文拉长」并不能真正扩展,需要新思路。整体保持克制:这是预览,不是炒作。As the course nears its end, this chapter glances at capabilities that are 'coming down the pipe.' First, models will rapidly go multimodal: not just text, but natively hearing and speaking (audio) and seeing and painting (images). In principle nothing fundamental changes — you tokenize audio (slices of the spectrogram) and images (patches) too, drop those tokens into the context window, and train with the same approach. Second, agents: today we still hand a single task to the model on a silver platter; in the future models will act as agents that coherently, error-correctingly carry out long, multi-step jobs — taking tens of seconds, minutes, or even hours, under human supervision (by analogy with the factory 'human-to-robot ratio,' the digital world will have a 'human-to-agent ratio'). Third, models will become more pervasive and invisible, integrated into tools (your computer / IDE), and will begin to take actions on your behalf (keyboard and mouse). Fourth, much research remains, e.g. test-time learning / 'thinking': today a deployed model's parameters are fixed, and the only 'learning' is in-context learning inside the context window; but the context window is a finite, precious resource that balloons for multimodal, long-running tasks, and simply 'making the context longer' won't truly scale, so new ideas are needed. The tone stays grounded: this is a preview, not hype.
  21. 21 去哪里找到这些模型Where to Find LLMs 一份实用的「找模型」导航。讲完模型是怎么炼成的,问题就变成:你究竟去哪儿才能用上它们?分四种情况。第一,最强的专有(proprietary)前沿模型——直接去模型提供方自己的网站,比如 OpenAI 的 ChatGPT。第二,想知道「此刻谁最强」——去排行榜,比如 LM Arena 排行榜,再配合一个 AI 新闻聚合站(如 AI News)追踪最新进展。第三,开放权重(open-weights)模型(如 DeepSeek、Llama)——去推理服务商(inference provider),比如 Together.ai,在它的 playground 里挑模型、直接对话。第四,想在自己电脑上本地跑——用 LM Studio 这类工具,加载更小的、被蒸馏(distilled)过、用更低精度的版本,就能塞进笔记本里离线运行。A practical guide to actually finding and using these models. Once you know how they're made, the question becomes: where do you go to use them? It splits into four cases. First, the strongest proprietary frontier models — go straight to the provider's own website, e.g. OpenAI's ChatGPT. Second, to know who is currently best — check a leaderboard such as the LM Arena leaderboard, paired with an AI-news aggregator (like AI News) to track the latest. Third, for open-weights models (e.g. DeepSeek, Llama) — go to an inference provider such as Together.ai, where you can pick a model in the playground and talk to it directly. Fourth, to run a model locally on your own machine — use a tool like LM Studio to load smaller, distilled, lower-precision versions that fit on a laptop and run offline.
  22. 22 全课总结Grand Summary 整门课的收束。我们回到最初的问题:当你在 ChatGPT 里敲下一句话、点击发送,屏幕背后到底发生了什么?把整条流水线串成一条线:预训练(互联网 → token → 下一个 token 预测 → base 模型,知识从互联网内化进网络参数)→ 后训练。后训练又分三步:SFT(用人类标注员撰写的理想对话做模仿学习,模型的「人格」就来自这里)→ 在可验证领域里的强化学习(RL,让模型在大量练习题上自己摸索出好的思考策略)→ 面向不可验证领域的 RLHF(用人类偏好排序训出一个奖励模型来引导)。于是你在 ChatGPT 里对话的对象,本质是一个 token 模拟器:它先模仿人类标注员,再被 RL 塑形。它强大但并不完美,带着锋利的边缘——会幻觉、有「瑞士奶酪」式的参差智能、连数字符也会出错。所以把它当工具:核对它的产出,保留「人在回路」。这是 22 章课程的终点,以鼓励作结。The close of the whole course. We return to the opening question: when you type into ChatGPT and hit go, what is actually happening behind the screen? We string the whole pipeline into one line: pretraining (internet -> tokens -> next-token prediction -> base model, with knowledge internalized from the internet into the network's parameters) -> post-training. Post-training has three steps: SFT (imitation learning on ideal conversations written by human labelers — this is where the model's 'personality' comes from) -> reinforcement learning in verifiable domains (RL, where the model practices on many problems and discovers good thinking strategies on its own) -> RLHF for unverifiable domains (training a reward model from human preference rankings to guide it). So what you talk to in ChatGPT is essentially a token simulator: it first imitates a human labeler, then is shaped by RL. It is powerful but imperfect, with sharp edges — it hallucinates, has 'Swiss-cheese' jagged intelligence, and even miscounts characters. So treat it as a tool: verify its work and keep a human in the loop. This is the end of the 22-chapter course, and we close on encouragement.