分词:文本如何变成 token Tokenization: How Text Becomes Tokens

在把文本送进神经网络之前,要先决定如何表示它。这类网络期望的是一个一维的符号序列,而且符号的种类是有限的。所以问题变成:我们的「符号」到底是什么?文本本身已经是一维序列(从左到右、从上到下);在计算机里,用 UTF-8 编码后它就变成了一长串比特,只有 0 和 1 两种符号。 Before feeding text into a neural network, we must decide how to represent it. These networks expect a one-dimensional sequence drawn from a finite set of symbols. So the question becomes: what are our symbols? Text is already a one-dimensional sequence (left to right, top to bottom); inside a computer, UTF-8 encoding turns it into a long string of bits, with only two possible symbols, 0 and 1.

问题是:序列长度在神经网络里是一种「珍贵且有限」的资源。我们并不想要一个只有两种符号、却极其漫长的序列。于是我们在「符号种类(词表大小)」和「序列长度」之间做权衡:宁愿多一些符号、换来短一些的序列。最朴素的压缩办法是把连续的 8 个比特打包成一个「字节」——8 个比特共有 256 种组合,所以序列瞬间缩短为原来的八分之一,代价是符号种类从 2 变成 256(每个值 0–255)。 The catch: sequence length is a precious, finite resource inside a neural network. We don't want a sequence with only two symbols that is extremely long. So we trade off the number of symbols (the vocabulary size) against sequence length: we'd rather have more symbols in exchange for a shorter sequence. The most naive compression is to pack 8 consecutive bits into one 'byte' — 8 bits have 256 possible combinations, so the sequence instantly becomes 8x shorter, at the cost of growing from 2 symbols to 256 (each value 0–255).

对最先进的模型来说,我们还想继续缩短序列、换取更多符号,做法是运行「字节对编码(Byte Pair Encoding,BPE)」。它的思路是:找出序列里非常常见的相邻符号对,比如某两个字节经常挨在一起出现(例如 116 后面跟着 32),就把这一对合并成一个全新的符号(铸造一个新 ID,例如 256),并把序列中所有这一对都改写成新符号。这个过程可以反复迭代,每铸造一个新符号,序列就更短一点、词表就更大一点。 For state-of-the-art models we want to shrink the sequence further in exchange for more symbols, and the way to do this is to run Byte Pair Encoding (BPE). The idea: find very common adjacent symbol pairs — say two bytes that frequently occur together (for example 116 followed by 32) — and merge that pair into a brand-new symbol (mint a new ID, e.g. 256), rewriting every occurrence of that pair as the new symbol. This can be iterated as many times as we like; each new symbol makes the sequence a bit shorter and the vocabulary a bit larger.

实践中一个不错的词表大小大约是 10 万个符号;具体来说,GPT-4 使用 100,277 个符号(其分词器叫 cl100k_base)。把原始文本转换成这些符号(我们称之为 token)的整个过程,就叫分词(tokenization)。下面这张图可以帮你直观感受这条「比特 → 字节 → BPE」的路径:每往右走一步,序列变短、词表变大。 In practice a good vocabulary size is around 100,000 symbols; specifically, GPT-4 uses 100,277 symbols (its tokenizer is called cl100k_base). The whole process of converting raw text into these symbols — which we call tokens — is tokenization. The diagram below helps you feel this 'bits → bytes → BPE' path: each step to the right makes the sequence shorter and the vocabulary larger.

现在用真实的 GPT-4 分词器亲手试一试。在下面的演示里输入文本,你会看到它被切成一个个带 ID 的 token。试着对比:「hello world」是两个 token;但「helloworld」、首字母大写的「Hello world」、或在中间多打一个空格,切分结果都会不同——token 数量甚至每个 token 本身都会变。 Now try it hands-on with the real GPT-4 tokenizer. Type text into the demo below and watch it get split into tokens, each with an ID. Compare for yourself: 'hello world' is two tokens; but 'helloworld', a capitalized 'Hello world', or an extra space in the middle all tokenize differently — the token count, and even the individual tokens, change.