Tokenization and BPE

Training BPE on the catalog

This phase uses a corpus, a collection of text for training. It holds 300 short product descriptions for a made-up outdoor brand. The code below trains BPE on it, with 200 merges. It works like the worked example, with two extra steps.

• First it splits the text into words, numbers and punctuation. That way, a merge never glues two words together. This step is called pre-tokenization.
• Each word keeps its leading space, so " layer" and "layer" are different. GPT-style tokenizers do the same.

import re
from collections import Counter
import requests

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
pattern = r" ?[A-Za-z]+| ?[0-9]+| ?[^\sA-Za-z0-9]+|\n"          # split into words first
word_counts = Counter(re.findall(pattern, text))
splits = {w: list(w) for w in word_counts}                     # every word starts as characters
merges = []

for step in range(200):
    pairs = Counter()
    for w, n in word_counts.items():
        for a, b in zip(splits[w], splits[w][1:]):
            pairs[a, b] += n                                   # count adjacent pairs, weighted
    best = max(pairs, key=pairs.get)
    merges.append(best)
    for w in word_counts:                                      # apply the merge everywhere
        s, out, i = splits[w], [], 0
        while i < len(s):
            if i < len(s) - 1 and (s[i], s[i + 1]) == best:
                out.append(s[i] + s[i + 1]); i += 2
            else:
                out.append(s[i]); i += 1
        splits[w] = out

print("first merges:", [a + b for a, b in merges[:12]])
print("merges 96–105:", [a + b for a, b in merges[95:105]])

def encode(s):
    tokens = []
    for w in re.findall(pattern, s):
        piece = list(w)
        for a, b in merges:                                    # replay merges in learned order
            out, i = [], 0
            while i < len(piece):
                if i < len(piece) - 1 and (piece[i], piece[i + 1]) == (a, b):
                    out.append(a + b); i += 2
                else:
                    out.append(piece[i]); i += 1
            piece = out
        tokens += piece
    return tokens

for s in ["It is light and fully waterproof.", "Our marketing team loves spreadsheets."]:
    t = encode(s)
    print(f"\n{len(s)} characters → {len(t)} tokens: {t}")
total = sum(len(splits[w]) * n for w, n in word_counts.items())
print(f"\nwhole corpus: {len(text):,} characters → {total:,} tokens ({len(text) / total:.1f} characters per token)")

Here's what the code does.

1. It downloads the catalog, splits it into words and counts how often each word appears.
2. It turns every word into a list of single characters.
3. It repeats 200 times: count every neighboring pair, weighted by how often its word appears. Then merge the most frequent pair everywhere, and record the merge.
4. encode tokenizes new text. It splits the text into words, then replays the learned merges in the order they were learned.
5. Last, it compares characters with tokens, for two test sentences and for the whole catalog.

The first merges are short, frequent fragments, such as " f", "ad" and "er". Later merges are whole words from the catalog, such as " layer" and " easy". Text that looks like the catalog shrinks to few tokens. The sentence about spreadsheets never appears in the catalog. So it falls apart into many small pieces.

Real tokenizers

Real tokenizers follow the same recipe, with a few upgrades.

They start from bytes, not characters. Computers store text as bytes: small numbers from 0 to 255. UTF-8 is the standard rule for turning text into bytes. A plain English letter takes one byte. "é" takes two, and many emoji take four. Byte-level BPE starts from all 256 possible bytes instead of from characters. GPT-2 and later GPT models use it, and so do many open models. It can encode any input at all: any language, emoji, or odd binary junk. In the worst case, it spells the text one byte at a time.

Their vocabularies are big. A real vocabulary typically holds 50,000 to 200,000 tokens.

They pre-tokenize. They split text into words, numbers and punctuation first, as the catalog code did. So merges never cross word boundaries.

They add special tokens. These are added by hand, outside the merging. Examples are a marker for "end of text" or for "the user is speaking now". Chat formats are built from them.

Other methods exist. SentencePiece is a popular tool that can train BPE or a method called Unigram. Unigram works the other way around. It starts with a large vocabulary and uses probabilities to drop the least useful pieces. The aim is the same: frequent text gets short codes.

Why tokenization explains odd LLM behavior

• Cost and context. Large tokenizers trained mostly on English average roughly four characters per token on English prose. Some languages, code styles and formats were rare in the tokenizer's training data. They need more tokens for the same content. So they cost more, and they fill the context window faster.
• Spelling and counting letters. A model sees " waterproof" as one or two token IDs, not as ten letters. So "How many r's are in strawberry?" asks about something the model never sees directly.
• Numbers. Digits can be grouped unevenly. "2025" might be one token while "2026" is two. That's one reason arithmetic is harder for a model than it looks.
• Leading spaces. " Summit" and "Summit" are different tokens, with different IDs.

For marketers: brand names, product codes and campaign hashtags often split into several tokens. That's harmless for meaning. But it explains why token counts, and costs, can run higher than a word count suggests for catalog or multilingual content.