Ask a chat model how many r's are in "strawberry" and you may get two. Ask it to spell a rare name backwards and watch it stumble. People read this as stupidity, which is the wrong lesson: the same model writes fluent paragraphs about strawberries. The right lesson is that the model never sees letters. It receives token IDs, and the vector it learns for each ID is shaped by the company the token keeps, not by its spelling.
Knowing this tells you what to hand the model and what to keep for code. Character-level work (counting, reversing, checking a format) belongs in a tool the model calls; the model's job is deciding to call it. It also explains why your bill and your context limit are counted in tokens, and why spelling a word out with spaces suddenly makes a model better at reading it.
What you'll be able to do
Show what a model receives in place of a word, and why a word it has never seen becomes one unknown token
Read what an embedding carries by looking at a word's nearest neighbors
Probe an embedding for letter information and interpret a probe that does worse than guessing
Explain why large models spell only a little, and what to do instead for character-level tasks
Quick check
A chat model is asked how many r's are in "strawberry" and says two. A guess at the cause:
The training data contained a typo that the model memorized as the spelling
The model never sees letters; it sees token IDs, and nothing about spelling comes with them
The model reads the word as ASCII codes and adds them up wrongly when a letter repeats
The model counts the letters one by one but loses count on words longer than eight letters
Show the answer
The model never sees letters; it sees token IDs, and nothing about spelling comes with them
A tokenizer replaces each piece of text with an integer before the model runs. The model's vector for that integer is learned from context, not from the characters. Counting letters means recalling spelling facts it was never directly taught.
The idea: the model never sees letters
Purpose. What a model receives in place of a word, and why that hides the letters.
In Tokens and the context window you saw that a prompt reaches a model as a list of integers, one per token, and in Tokenization and BPE you followed "The cat sat" through the model as numbers. This lesson asks what that implies for a question like "how many r's in strawberry?".
The answer starts at the first step. A tokenizer looks each piece of text up in a fixed vocabulary. A common word is one token; a rare word is several pieces; and the model gets only the IDs. Whatever it comes to know about a token, it has to learn from the contexts the token appears in, because nothing else is ever shown to it. The course's tiny GPT makes this stark: its tokenizer is word-level, with 195 entries, so a word it has never met is not even split into pieces. It becomes the single unknown token.
Worked example
What the model receives
The tiny GPT's tokenizer on four inputs. The unknown token's ID is 3.
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
print("vocabulary size:", len(tok.vocab), " unknown token id:", tok.unk_id)
for text in ["tent", "Tent", "strawberry", "Our Kestrel tent is made from"]:
print(f"{text!r:>33} -> {tok.encode(text)}")
It runs right here in your browser, with nothing to install.
"tent" is token 174. "Tent" with a capital letter is a different string, never seen in the catalog, so it becomes 3, the unknown token, exactly like "strawberry". The last line is what a whole prompt looks like from the inside: six integers. Everything the model will do with "tent", it does with the number 174, and it has no way to see that 174 is spelled t-e-n-t.
Quick check
A word-level tokenizer with an unknown ID of 3. What does this print?
vocab = {"tent": 174, "mug": 120}
ids = [vocab.get(w, 3) for w in"tent Tent mug".split()]
print(ids, len(set(ids)))
[174, 3, 120] 3
[174, 3, 120] 2
[174, 174, 120] 2
[3, 3, 3] 1
Show the answer
[174, 3, 120] 3
"Tent" is not in the vocabulary, so it falls back to 3. Three different IDs come out, and the model will treat "Tent" as a stranger even though you can see it is the same word.
What a token carries
Inside the model, token 174 selects row 174 of the embedding matrix: 32 numbers for the tiny GPT, a few thousand for a production model. Those numbers were set by gradient descent, pushed around on every training step so that the model's predictions improve. The signal that shapes them is the prediction loss on the surrounding tokens. A word's row therefore ends up encoding the company it keeps: which tokens come before and after it, what it predicts, what predicts it.
Nothing in that signal mentions spelling. Two words that appear in the same sentences get similar rows even if they share no letters; two words that share every letter but one can be far apart. You can see the kind of information that is there by asking which rows are closest to a word's row, measured by the cosine of the angle between them (Angle, cosine similarity and projection).
Worked example
A word's neighbors
For two words, the four vocabulary entries whose embedding rows point in the most similar direction.
import re
import numpy as np
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
E = model.tok.weight.detach().numpy()
words = [w for w in tok.vocab if re.fullmatch(r"[a-z]+", w)]
defneighbors(word, k=4):
v = E[tok.token_id(word)]
sims = []
for other in words:
if other == word:
continue
u = E[tok.token_id(other)]
sims.append((float(v @ u / (np.linalg.norm(v) * np.linalg.norm(u))), other))
return", ".join(f"{w} {s:.2f}"for s, w insorted(sims, reverse=True)[:k])
for word in ["tent", "weekends"]:
print(f"{word:<9} {neighbors(word)}")
In the lesson you can change this and run it.
"tent" sits next to "pitch", "roomy", "packable" and "breathable": the words that describe tents in the catalog ("quick to pitch", "light and roomy"). "weekends" sits next to "coffee" and "nights", from phrases like "slow weekends" and "cold nights in camp". The rows encode usage. Nothing here says that "tent" has four letters or that "weekends" ends in s.
Try it now
The widget's split is approximate, like a real tokenizer's: the model gets pieces, never letters.
In your own words
In your own words: what decides the numbers in a token's embedding row, and why would two words that are spelled alike not end up near each other?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: probe the embedding for letters
Purpose. Fit two probes on the embedding rows and compare each with the guess that needs no model.
If letter information were in the rows anyway, as a side effect, you could read it out: fit a small linear model from a row's 32 numbers to the word's letter count, on part of the vocabulary, and test it on the rest. This is a probe, the same tool the interpretability lesson used to ask what a hidden state contains.
The baseline matters as much as the probe. A probe that cannot beat "guess the average length for every word" has found nothing. The next cell fits two probes with least squares: one for the letter count, one for whether the word contains the letter e.
Worked example
Two probes, two baselines
136 lowercase words, 95 for fitting and 41 held out. Least squares from the embedding row to the letter count, and to a yes/no for "contains e".
import re
import numpy as np
from tinygpt import load_pretrained
model, tok = load_pretrained("tiny-gpt")
E = model.tok.weight.detach().numpy()
words = [w for w in tok.vocab if re.fullmatch(r"[a-z]+", w)]
rows = np.array([E[tok.token_id(w)] for w in words])
X = np.concatenate([rows, np.ones((len(words), 1))], axis=1) # a bias column
order = np.random.default_rng(0).permutation(len(words))
train, test = order[:95], order[95:]
length = np.array([float(len(w)) for w in words])
coef = np.linalg.lstsq(X[train], length[train], rcond=None)[0]
probe_error = np.abs(X[test] @ coef - length[test]).mean()
guess_error = np.abs(length[train].mean() - length[test]).mean()
print(f"letter count: probe off by {probe_error:.2f} letters on average, guessing the "f"mean off by {guess_error:.2f}")
has_e = np.array([1.0if"e"in w else0.0for w in words])
coef = np.linalg.lstsq(X[train], has_e[train], rcond=None)[0]
probe_acc = np.mean((X[test] @ coef > 0.5) == (has_e[test] > 0.5))
majority = max(has_e[train].mean(), 1 - has_e[train].mean())
print(f"contains an e: probe right {probe_acc:.0%} of the time, always guessing the "f"majority {majority:.0%}")
import matplotlib.pyplot as plt
fig, (left, right) = plt.subplots(1, 2, figsize=(8, 3))
left.bar(["probe", "guess the mean"], [probe_error, guess_error])
left.set_title("Letter count: error in letters")
right.bar(["probe", "guess the majority"], [probe_acc, majority])
right.set_title("Contains e: share right")
plt.show()
In the lesson you can change this and run it.
Both probes lose to their baselines. The letter-count probe is off by 2.16 letters on average where guessing the mean is off by 1.77; the "contains e" probe is right 44% of the time where the majority guess gets 54%. A linear readout can find nothing about spelling in these rows, because nothing about spelling was ever put there. The rows are rich in another way, as the neighbors showed.
Quick check
A probe for a word's letter count is off by 2.16 letters on held-out words, while guessing the average length is off by 1.77. What is the right reading?
The baseline is wrong because the mean is not a prediction a model would make
The embedding encodes letter count, but a linear probe is too simple to decode it
The embedding carries no usable letter-count information; the probe fitted noise
The probe proves the model counts letters internally but hides the result
Show the answer
The embedding carries no usable letter-count information; the probe fitted noise
Beating the baseline is the minimum. A probe that does worse on held-out words learned the training words' quirks and nothing general, which is what you expect when the property was never part of the training signal.
Drill
Letters from pieces
Write letters_from_pieces(pieces, table, letter). table maps each piece to a dictionary of letter counts, like {"straw": {"s": 1, "t": 1, "r": 1, "a": 1, "w": 1}}. Return the total count of letter across the pieces, treating a letter that a piece's dictionary does not list as 0.
In the lesson, your code is checked as you write it.
Why big models can spell a little
Production tokenizers are not word-level. Byte-pair encoding splits rare words into a few pieces, so "strawberry" might arrive as "straw" and "berry", and a made-up word as five or six fragments. That helps a little: a model can learn facts about frequent pieces. Training text does sometimes spell things out, in acronyms, lists, dictionaries, code and children's books, and from those passages a model picks up a partial table of spelling facts per piece. Counting a letter then means recalling the right facts for each piece and adding them, the chain in the stepper, with a chance to slip at every link.
That is why the behavior is inconsistent rather than absent. Common pieces are spelled well, rare ones badly; a word split one way is counted right and the same word in a different context, split differently, is not. The tokens lesson already explained the other consequences of pieces: cost, context limits and why rare words are expensive. Here the consequence is a blind spot for anything that lives below the piece.
Quick check
A production model counts the r's in "berry" correctly but miscounts them in "strawberry". Which explanation fits the mechanism?
Longer words take more time, and the model runs out before it reaches the last letters
One familiar piece carries reliable spelling facts; several pieces need every fact right and added
Capital letters in the training data confused it about where words begin
The model knows the fruit but not the compound word, so it falls back to a guess
Show the answer
One familiar piece carries reliable spelling facts; several pieces need every fact right and added
Counting runs through pieces. One frequent piece can carry a reliable spelling fact; a word split into several pieces needs every fact to be right and the addition to be right too.
Quick check
From the tokens lesson: why does a rare or misspelled word cost more tokens than a common one?
The model charges extra for words it has to look up outside its vocabulary
Rare words are stored in a slower part of the vocabulary that costs more to read
Its pieces were never merged into one token, so it stays as several short tokens
Common words are free, so only the rare ones show up on the bill
Show the answer
Its pieces were never merged into one token, so it stays as several short tokens
BPE merges the pairs that occur most often in the training text. A rare word's pieces never earned a merge, so it stays in fragments, each a token you pay for and that the model has to interpret.
Drill
Split a word into pieces
Write pieces_for(word, vocab): split word into pieces from vocab by always taking the longest piece that matches at the current position, left to right. If no piece matches, take the single character as a piece.
In the lesson, your code is checked as you write it.
In your own words
Why can a production model count the letters in a common word more reliably than in a rare one?
Write a sentence or two, then compare it with a good answer in the lesson.
In production: what this justifies
Purpose. What to hand to code, what to change in the tokens, and what to measure.
Hand character-level work to code. Counting, reversing, checking a serial number's format, comparing two spellings: a line of Python does each exactly, and a tool-using model (the tools lesson) only has to decide to call it. Asking the model to do it in its head asks for a chain of recalled spelling facts.
When you cannot use a tool, change the tokens. Spell the word out with spaces, "s t r a w b e r r y", and every letter becomes its own token, something the model can attend to directly. Models count spaced letters far better than letters hidden inside a piece. The same move helps with codes, part numbers and names.
And measure in tokens. The context limit, the bill and the speed are all per token, so a text full of rare words or numbers is longer than it looks. The tokens lesson's counting tools are the right instruments; a word count is not.
Worked example
Code counts; spacing makes letters visible
Three ways to get at the letters in a word: directly in Python, through pieces, and by spacing the word so each letter stands alone. The pieces come from the greedy splitter above.
defpieces_for(word, vocab):
pieces, i = [], 0while i < len(word):
best = word[i]
for j inrange(len(word), i, -1):
if word[i:j] in vocab:
best = word[i:j]
break
pieces.append(best)
i += len(best)
return pieces
vocab = {"straw", "berry", "st", "raw", "aw"}
word = "strawberry"print("python:", word.count("r"), "r's in", len(word), "letters")
print("as pieces:", pieces_for(word, vocab), "-> the model sees",
len(pieces_for(word, vocab)), "tokens and no letters")
spaced = " ".join(word)
print("spaced:", spaced, "->", len(spaced.split()), "tokens, one per letter;",
spaced.split().count("r"), "of them are r")
In the lesson you can change this and run it.
Python answers in one call. As pieces, the word is two tokens with the letters sealed inside them. Spaced out, it is ten tokens, each a single letter, and a model can attend to each one; that is why the spelling trick works and why serial numbers read better with separators. The tool and the spacing both do the same thing: they move the letters from inside a token to where the model can see them.
Quick check
What does this print?
word = "strawberry"print(word.count("r"), " ".join(word).split().count("r"), len(" ".join(word).split()))
2 3 10
3 10 10
3 3 2
3 3 10
Show the answer
3 3 10
Counting the letter directly gives 3; spacing the word out and counting tokens equal to "r" also gives 3, out of 10 single-letter tokens. Code sees letters; a model sees whatever tokens you give it.
Where this shows up in an engineer's week
A bug report says the model mangles serial numbers. Tokenize a few real ones and look at the pieces. If a code is split into odd fragments, validate it with a regular expression or a checksum in code, and show the model only what it needs to decide. Measure prompts in tokens, since cost and context limits are per token.
11DistillationTrain a small model to imitate a large one.35 min
12Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
13QuantizationStore weights in fewer bits to run models cheaply.35 min
23Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
24Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
25Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
26Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
32Weight-space programming: what it is, and what has been shownFreeOptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
33Anatomy of a checkpointOptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
34Reading weights without running the modelOptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
35Writing weights by handOptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
36Edits are deltas: task vectors, patches and hashesOptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
37Merging modelsOptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
38Low-rank editsOptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
39Directions: steer the run or edit the weightsOptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
40Editing one factOptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
41The editor's harnessOptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
42An LLM reads a network's weightsOptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
43An LLM fixes a model's weightsOptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
44An LLM writes a network's weightsOptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
45Did it work? Testing an editOptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
46The published edit, on a real modelOptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
47Safety, licenses and provenanceOptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
48Capstone: ship a custom modelOptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write