推理:从模型生成文本 Inference: Generating Text from the Model

我们已经看过网络的内部结构,也大致谈了训练。现在来看处理这些网络的另一个主要阶段:推理(inference)。在推理中,我们要做的是从模型生成新数据,从而看看它在参数里到底内化了哪些模式。从模型生成其实相当直接:先准备一些起始 token,也就是你想要的「前缀」。 We've seen the network's internals and discussed training a bit. Now for the other major stage of working with these networks: inference. In inference, what we do is generate new data from the model, so we can see what patterns it has internalized in its parameters. Generating from the model is fairly straightforward: we start with some tokens that form your 'prefix' — whatever you want to begin with.

假设我们想以 token 91 开头。把它喂进网络,网络会像之前说的那样,给出一个概率向量。接下来我们就「掷一枚有偏的硬币」——根据这个概率分布采样出一个 token。模型给了高概率的 token,被采到的可能性就更大;你可以这样理解这枚有偏的硬币。比如这次采样得到 token 860,它跟在 91 后面是个相对可能的选择。然后我们把 860 接到序列末尾,再问「第三个 token 是什么」,采样、追加;如此循环。下面这张图就展示了这个生成循环。 Say we want to start with token 91. We feed it into the network, and the network gives us a probability vector, just as before. Then we 'flip a biased coin' — we sample a token according to this probability distribution. Tokens the model assigned high probability are more likely to be sampled; that's the biased coin. Say this time we sample token 860, a relatively likely continuation of 91. We append 860 to the sequence, then ask 'what's the third token?', sample, append, and loop on. The diagram below shows this generation loop.

在上面这个例子里,我们其实没有完全复现训练数据里的那段序列。比如某一步我们采到的不是训练里原本的 3962,而是 13659(对应词「article」)——于是后面的走向就和原文不同了。记住:这些系统是随机的,我们在采样、在掷硬币。有时运气好,会复现训练集里的一小段文本;有时采到的 token 压根没在任何训练文档里逐字出现过。 In the example above, we didn't exactly reproduce the sequence from the training data. At one step we sampled 13659 (the word 'article') instead of the original 3962 — and from there the continuation diverged from the source. Remember: these systems are stochastic; we're sampling, we're flipping coins. Sometimes we get lucky and reproduce a small chunk of training text; sometimes we sample a token that never appeared verbatim in any training document.

所以我们得到的是训练数据的「remix」。因为每一步都可能掷出略微不同的 token,而那个 token 一旦进入序列,又会影响下一步的采样,于是序列很快就会偏离任何训练文档。统计上,生成文本与训练数据有相似的性质,但并不与之完全相同——它更像是「受训练数据启发」而来。至于为什么会采到「article」?你可以想象,在「viewing a single …」这样的上下文里,「article」本就是个相对可能的词,训练文档里某处也确实这样接续过,我们只是恰好在这一步把它抽了出来。 So what we get is a 'remix' of the training data. Because each step can flip out a slightly different token, and once that token enters the sequence it influences the next sample, the sequence quickly drifts away from any training document. Statistically, the generated text has similar properties to the training data but is not identical to it — it's more like text 'inspired by' the training data. As for why we'd sample 'article'? You can imagine that in a context like 'viewing a single ...', 'article' is a relatively likely word, and somewhere in the training documents it did follow such a context — we just happened to draw it at that step.

下面这个演示让你亲手体验:给定上下文,网络给出下一个 token 的概率分布,你来「掷硬币」采样。多采几次,留意同样的前缀如何导向不同的 token——这正是推理随机性的来源。 The demo below lets you experience this firsthand: given a context, the network gives a probability distribution over the next token, and you 'flip the coin' to sample. Sample a few times and notice how the same prefix can lead to different tokens — that's exactly where inference's randomness comes from.