Index Ai Academy
Phase 5 · How transformers think

Tokenization: text to integer IDs

Follow "The cat sat" into the three integers a model receives.

About 30 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

A language model never reads letters. The first thing that happens to your prompt is a lookup: each piece of text becomes a whole number, and the model receives a short list of those numbers. Everything after that, the embeddings, the attention and the final prediction, is arithmetic on that list. This section follows one prompt, "The cat sat", through the whole pipeline, and this lesson is its first step: the text becomes the three integers 104, 324 and 782.

Knowing this step well pays off every day. It explains why a model charges you per token rather than per word, why a misspelled word is expensive, why the same word can get two different IDs, and what the "context window" counts.

What you'll be able to do

  • Split a sentence into tokens with byte-pair merges, and read off each token's integer ID
  • State what V and T stand for, and give the shape of the tensor a prompt becomes
  • Predict why a rare word costs more tokens than a common one
  • Turn a list of IDs back into text, and say what the model never sees
"The cat sat" is cut into three tokens and looked up as 104, 324 and 782, then the boards show the sequence, V and T, and the byte-pair merges that built the vocabulary. Transcript
Quick check
A tokenizer is, at heart, a lookup table. A guess: what does this print?
vocab = {"The": 104, " cat": 324, " sat": 782}
print([vocab[t] for t in ["The", " cat", " sat"]])
  1. {104, 324, 782}
  2. ['The', ' cat', ' sat']
  3. [104, 324, 782]
  4. 3
Show the answer

[104, 324, 782]

Each token string maps to one integer, and the list keeps the order of the sentence. Those three numbers, in that order, are all the model will ever see of "The cat sat".

The idea: text becomes a list of integers

This rests on Tokenization and BPE, where byte-pair merges were learned from a text. Here one prompt, "The cat sat", starts its trip through the model, and the first step is the lookup.

What the model receives

Type "The cat sat" into a chat window and the model gets none of those letters. A small program called the tokenizer runs first. It cuts the text into pieces called tokens, and replaces each piece with its position in a fixed list, the vocabulary. The model receives only those positions: a one-dimensional list of integers.

Two numbers describe that list:

  • VV, the vocabulary size: how many different tokens exist. Our small model in this section has V=1000V = 1000. GPT-2 has 50,257; recent models have 100,000 to 250,000.
  • TT, the sequence length: how many tokens this prompt became. "The cat sat" has T=3T = 3.

In symbols, the prompt becomes a sequence x=(x1,x2,…,xT)x = (x_1, x_2, \ldots, x_T), where every xtx_t is a whole number from 0 to V−1V - 1. For our prompt that's x=(104,324,782)x = (104, 324, 782), a tensor of shape (T,)=(3,)(T,) = (3,). Nothing about the letters c, a, t survives. The number 324 means "cat" only because position 324 in the vocabulary happens to hold " cat".

The splitter below shows roughly where a real tokenizer cuts. Try a long word, a number or a misspelling and watch the count change.

Try it now

An approximate splitter: where a tokenizer tends to cut, and how many tokens a text costs. The real rule, byte-pair merges, is below.

Quick check
From the lesson on tokenization and BPE: which of these is GPT-2's vocabulary size, the V its tokenizer works with?
  1. 4,096
  2. 1,024
  3. 50,257
  4. 768
Show the answer

50,257

GPT-2's byte-level BPE vocabulary has 50,257 entries: 256 byte tokens, 50,000 merges and one end-of-text token. 768 is its embedding width and 1,024 its context length.

Byte-pair encoding, recapped

Where do the pieces come from? Most tokenizers in use today were made by byte-pair encoding (BPE), a short procedure run once over a large training text:

  1. Start with the smallest units. For GPT-2 those are the 256 possible bytes, which is why any text in any language can be encoded, with no "unknown" token.
  2. Count every adjacent pair of units in the training text.
  3. Merge the most frequent pair into one new unit, and add it to the vocabulary.
  4. Repeat step 2 and 3 until the vocabulary has the size you want.

The list of merges, in order, is the tokenizer. Common strings get merged early and become single tokens, so " the" or " cat" cost one token each. Rare strings never get merged and stay in small pieces, which is why an unusual name costs four or five tokens.

The example below runs six merges on a tiny training text. Read the printed order: at merges first because it appears five times (in cat, sat, mat, cat and ate), then th and the.

Worked example

Six merges on a tiny training text

Count adjacent pairs, merge the most frequent, repeat. The merges are printed in the order they're learned, with how often the pair occurred.

from collections import Counter

def count_pairs(words):
    pairs = Counter()
    for w in words:
        for a, b in zip(w, w[1:]):
            pairs[(a, b)] += 1
    return pairs

def merge(words, pair):
    out = []
    for w in words:
        i, new = 0, []
        while i < len(w):
            if i < len(w) - 1 and (w[i], w[i + 1]) == pair:
                new.append(w[i] + w[i + 1]); i += 2
            else:
                new.append(w[i]); i += 1
        out.append(new)
    return out

text = "the cat sat on the mat the cat ate"
words = [list(w) for w in text.split()]      # every word starts as its characters
for step in range(6):
    pairs = count_pairs(words)
    # the most frequent pair; ties broken alphabetically
    best = max(pairs, key=lambda p: (pairs[p], p))
    words = merge(words, best)
    print(step + 1, "merge", best, "seen", pairs[best], "times ->", repr("".join(best)))
print([" ".join(w) for w in words])
print("tokens:", sum(len(w) for w in words), " characters:", len(text.replace(" ", "")))
It runs right here in your browser, with nothing to install.

The first merge is ('a', 't'), seen 5 times. Then th (3 times), and the (the new unit th followed by e, 3 times). Then cat (2), and sat and on once each. After six merges the 26 characters of the text are 11 tokens: whole words where a word was frequent (the, cat, sat, on), and pieces where it wasn't (m at, at e). Real tokenizers do the same thing with tens of thousands of merges over gigabytes of text.

Quick check
In the example, why does at merge before th?
  1. Pairs are merged alphabetically, and a comes before t
  2. It occurs 5 times and th only 3; the most frequent pair merges first
  3. at is shorter to type
  4. Two-letter pairs that end a word merge first
Show the answer

It occurs 5 times and th only 3; the most frequent pair merges first

BPE always merges the pair it counted most. at appears in cat, sat, mat, cat and ate (5); th appears in the three thes (3). The alphabetical rule only breaks ties.

Drill

Count the adjacent pairs

Write count_pairs(words). words is a list of lists of pieces, like [['c', 'a', 't'], ['s', 'a', 't']]. Return a dictionary (a Counter is fine) from each adjacent pair (left, right) to how many times it occurs across all the words.

from collections import Counter

def count_pairs(words):
    pairs = Counter()
    return pairs

print(count_pairs([list("cat"), list("sat")]))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what does the model receive in place of your text, and which of V and T changes when the prompt gets longer?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: IDs, a tensor and the model's own tokenizer

From merges to IDs

Training happens once. At inference, when you type a prompt, the tokenizer only applies what it learned: it runs the merges over your text in the same order (which comes to the same thing as matching the longest piece it knows), and then looks each piece up in the vocabulary list. The ID is the piece's position in that list. Nothing more.

One convention surprises people: the space belongs to the token that follows it. " cat" with its leading space is one entry in the vocabulary, and "cat" at the very start of a text is a different entry with a different ID. That's why our prompt's tokens are "The", " cat" and " sat".

Here is a slice of our small model's vocabulary, with the three tokens we care about, and the lookup that turns a string into IDs. The period gets its own token, 13. (GPT-2 happens to use 13 for a period too: its first tokens are the printable characters in order, and "." is the 14th.)

Worked example

Encode a sentence with a vocabulary

Greedy longest match: at each position, take the longest piece the vocabulary knows, then move past it.

VOCAB = {"The": 104, " cat": 324, " sat": 782, " on": 511, " the": 262, " mat": 913,
         ".": 13, " ": 220, "a": 64, "c": 66, "s": 82, "t": 83}

def encode(text):
    ids, i = [], 0
    while i < len(text):
        for j in range(len(text), i, -1):      # try the longest piece first
            piece = text[i:j]
            if piece in VOCAB:
                ids.append(VOCAB[piece]); i = j; break
        else:
            raise ValueError(f"no token for {text[i]!r}")
    return ids

print(encode("The cat sat"))
print(encode("The cat sat on the mat."))
ID_TO_TOKEN = {v: k for k, v in VOCAB.items()}
print(repr("".join(ID_TO_TOKEN[i] for i in encode("The cat sat"))))
In the lesson you can change this and run it.

"The cat sat" becomes [104, 324, 782]: T=3T = 3. The longer sentence becomes seven IDs, the period included. Decoding is the lookup in reverse: join the pieces for each ID and you get the exact text back, spaces and all. A tokenizer loses nothing; it only changes the representation.

Quick check
Using the same VOCAB and encode(), how many IDs does "The cats sat" become?
VOCAB = {"The": 104, " cat": 324, " sat": 782, " on": 511, " the": 262, " mat": 913,
         ".": 13, " ": 220, "a": 64, "c": 66, "s": 82, "t": 83}

def encode(text):
    ids, i = [], 0
    while i < len(text):
        for j in range(len(text), i, -1):
            piece = text[i:j]
            if piece in VOCAB:
                ids.append(VOCAB[piece]); i = j; break
        else:
            raise ValueError(f"no token for {text[i]!r}")
    return ids

print(len(encode("The cats sat")))
  1. 2
  2. 3
  3. 5
  4. 4
Show the answer

4

" cats" isn't in the vocabulary, so the longest match is " cat" (324) and then the lone "s" (82): four IDs where "The cat sat" needed three. One extra letter, one extra token. This is what a misspelling or a rare word costs.

Drill

Decode IDs back to text

Write decode(ids, vocab) that returns the text for a list of IDs. vocab maps token strings to IDs, so you need the reverse lookup. decode([104, 324, 782], VOCAB) is "The cat sat".

VOCAB = {"The": 104, " cat": 324, " sat": 782, " on": 511, ".": 13}

def decode(ids, vocab):
    return ""

print(repr(decode([104, 324, 782], VOCAB)))
In the lesson, your code is checked as you write it.

The trained tiny GPT's own tokenizer

The vocabulary above, with V = 1000, is made up to keep the numbers small. The tiny GPT you will follow through the next lessons has a real one, with a different shape: it splits text into words, numbers and punctuation, and it was trained on a product catalog. Its encode does the lookup, and any word it never saw becomes the unknown token. The cell loads it and prints the vocabulary size, the pieces and the IDs for two prompts.

Worked example

Encode with the trained model's tokenizer

The first prompt uses words from the catalog. The second uses the lesson's sentence, which the catalog never contained.

from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
print("V =", len(tok), " unknown id =", tok.unk_id)
for text in ["Describe the Lark headlamp .", "The cat sat"]:
    ids = tok.encode(text)
    print(tok.tokenize(text), ids, " T =", len(ids))
print(repr(tok.decode(tok.encode("Describe the Lark headlamp ."))))
In the lesson you can change this and run it.

The tiny GPT has V = 195 tokens. "Describe the Lark headlamp ." becomes five IDs and decodes back to the same text. "The cat sat" becomes [51, 3, 3]: "The" is known, but "cat" and "sat" are not in a catalog of outdoor gear, so both turn into the unknown token 3. The model cannot tell them apart. That is the cost of a small vocabulary, and the reason real tokenizers use bytes: nothing is ever unknown.

Quick check
Using the tiny GPT's tokenizer, what does tok.encode("The dog sat") give, when only "The" is in its vocabulary?
  1. an error, since the vocabulary has no "dog"
  2. [51, 3]
  3. [51]
  4. [51, 3, 3]
Show the answer

[51, 3, 3]

Each piece the vocabulary does not hold becomes the unknown id 3, so three pieces give three ids: 51 for "The" and 3 twice.

One lookup, line by line

The step-through below is the encode loop from the first example on a three-piece vocabulary. Watch i move past each piece as the list of IDs grows.

The shape the rest of the model sees

From here on the model works with tensors, so the list of IDs becomes one: shape (T,)(T,), integer type. A chat server usually processes several prompts at once, stacked into a batch of shape (B,T)(B, T), padded to the same length. Every later step in this section keeps the TT rows, one per token: the embeddings will be T×dmodelT \times d_{\text{model}}, the attention matrix T×TT \times T, and the final prediction is made from the last row.

TT is also what a context window limits. A model with a 128,000-token window takes at most 128,000 IDs in one go, and that count includes the tokens of its own answer. Since tokens, not words, are the unit, text in English runs at about 1.3 tokens per word, code and other languages cost more.

The cell below uses the course's teaching version of PyTorch: the same code as the real library, on the CPU, much slower. Shapes and types come out the same.

Worked example

From a list of IDs to an integer tensor

The three IDs as the model receives them, then a batch of two prompts with the same length.

import torch

ids = torch.tensor([104, 324, 782])
print(tuple(ids.shape), ids.dtype)

batch = torch.tensor([[104, 324, 782],
                      [104, 324, 511]])   # "The cat sat" and "The cat on", side by side
print(tuple(batch.shape))
print("T =", batch.shape[1], " V =", 1000, " every ID below V:",
      bool((batch < 1000).all()))
In the lesson you can change this and run it.

One prompt is a tensor of shape (3,) with 64-bit integers. Two prompts stacked are (2, 3): B=2B = 2 sequences of T=3T = 3 tokens. Every ID must be below VV, because the next step uses it as a row number in a table with exactly VV rows.

Quick check
A server batches two prompts of three tokens each, as nested lists. What does this print?
batch = [[104, 324, 782], [104, 324, 511]]
print(len(batch), len(batch[0]))
  1. 6
  2. 3 2
  3. 2 6
  4. 2 3
Show the answer

2 3

The outer list has B = 2 prompts, each a list of T = 3 IDs: the (B, T) shape, read as two numbers.

Quick check
A prompt of 12 tokens enters a model with a vocabulary of 50,257. What is the shape of the tensor the tokenizer hands over?
  1. (12,)
  2. (50257,)
  3. (12, 50257)
  4. (1, 768)
Show the answer

(12,)

One integer per token: 12 tokens, shape (12,). The vocabulary size says how big each integer may be, not how many there are. The 50,257-wide and 768-wide shapes appear later, in the LM head and the embeddings.

In your own words
Why must every ID be below V before the model can use it?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: GPT-2's ids, and what they cost

The IDs in this section are made up so the arithmetic stays small. A real model's are not, and the same sentence gets different ones. The map shows where this step sits: the first two nodes, before any vector exists.

Where this shows up in an engineer's week

On the job: the token count of a prompt, not its word count, sets the bill and fills the context window. Count it with the provider's own counter before you size a batch or a budget, and log T next to every request: a jump in T is often the first sign that a template or a retrieval step started pasting too much.

Most tokenizer bugs are mismatches. A model trained with one tokenizer and served with another reads nonsense, with no error, because every ID is still a valid number. Pin the tokenizer's name and version beside the model's weights.

Goes deeper in

In your own words
Two models use different tokenizers. Why can't you feed one model the ids made for the other?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A tokenizer with a vocabulary of 50,257 turns a 9-word sentence into 11 IDs. Which statement is right?

  1. V = 50,257 and T = 11
  2. V = 11 and T = 50,257, since the IDs are the vocabulary
  3. V = 9 and T = 11, one vocabulary entry per word
  4. T = 50,257, the number of possible tokens, and V = 11

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 5 lessons

How Transformers Think: an interactive inference walkthrough

Follow one prompt, "The cat sat", through a language model with every number shown: tokens to IDs, embeddings and positional waves, the attention matrix, the feed-forward layer, and the probability over the next word.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write