Index Ai Academy
Phase 6 · Why models do that

Why it can't spell or count letters

The model sees token IDs, and an embedding learned from usage carries nothing about spelling.

About 25 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers9

Why it matters

Ask a chat model how many r's are in "strawberry" and you may get two. Ask it to spell a rare name backwards and watch it stumble. People read this as stupidity, which is the wrong lesson: the same model writes fluent paragraphs about strawberries. The right lesson is that the model never sees letters. It receives token IDs, and the vector it learns for each ID is shaped by the company the token keeps, not by its spelling.

Knowing this tells you what to hand the model and what to keep for code. Character-level work (counting, reversing, checking a format) belongs in a tool the model calls; the model's job is deciding to call it. It also explains why your bill and your context limit are counted in tokens, and why spelling a word out with spaces suddenly makes a model better at reading it.

What you'll be able to do

  • Show what a model receives in place of a word, and why a word it has never seen becomes one unknown token
  • Read what an embedding carries by looking at a word's nearest neighbors
  • Probe an embedding for letter information and interpret a probe that does worse than guessing
  • Explain why large models spell only a little, and what to do instead for character-level tasks
Quick check
A chat model is asked how many r's are in "strawberry" and says two. A guess at the cause:
  1. The training data contained a typo that the model memorized as the spelling
  2. The model never sees letters; it sees token IDs, and nothing about spelling comes with them
  3. The model reads the word as ASCII codes and adds them up wrongly when a letter repeats
  4. The model counts the letters one by one but loses count on words longer than eight letters
Show the answer

The model never sees letters; it sees token IDs, and nothing about spelling comes with them

A tokenizer replaces each piece of text with an integer before the model runs. The model's vector for that integer is learned from context, not from the characters. Counting letters means recalling spelling facts it was never directly taught.

The idea: the model never sees letters

Purpose. What a model receives in place of a word, and why that hides the letters.

This lesson rests on Tokenization and BPE, where text becomes token ids, and on Angle, cosine similarity and projection, which measures how close two vectors point.

In Tokens and the context window you saw that a prompt reaches a model as a list of integers, one per token, and in Tokenization and BPE you followed "The cat sat" through the model as numbers. This lesson asks what that implies for a question like "how many r's in strawberry?".

The answer starts at the first step. A tokenizer looks each piece of text up in a fixed vocabulary. A common word is one token; a rare word is several pieces; and the model gets only the IDs. Whatever it comes to know about a token, it has to learn from the contexts the token appears in, because nothing else is ever shown to it. The course's tiny GPT makes this stark: its tokenizer is word-level, with 195 entries, so a word it has never met is not even split into pieces. It becomes the single unknown token.

Worked example

What the model receives

The tiny GPT's tokenizer on four inputs. The unknown token's ID is 3.

from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
print("vocabulary size:", len(tok.vocab), " unknown token id:", tok.unk_id)
for text in ["tent", "Tent", "strawberry", "Our Kestrel tent is made from"]:
    print(f"{text!r:>33} -> {tok.encode(text)}")
It runs right here in your browser, with nothing to install.

"tent" is token 174. "Tent" with a capital letter is a different string, never seen in the catalog, so it becomes 3, the unknown token, exactly like "strawberry". The last line is what a whole prompt looks like from the inside: six integers. Everything the model will do with "tent", it does with the number 174, and it has no way to see that 174 is spelled t-e-n-t.

Quick check
A word-level tokenizer with an unknown ID of 3. What does this print?
vocab = {"tent": 174, "mug": 120}
ids = [vocab.get(w, 3) for w in "tent Tent mug".split()]
print(ids, len(set(ids)))
  1. [174, 3, 120] 3
  2. [174, 3, 120] 2
  3. [174, 174, 120] 2
  4. [3, 3, 3] 1
Show the answer

[174, 3, 120] 3

"Tent" is not in the vocabulary, so it falls back to 3. Three different IDs come out, and the model will treat "Tent" as a stranger even though you can see it is the same word.

What a token carries

Inside the model, token 174 selects row 174 of the embedding matrix: 32 numbers for the tiny GPT, a few thousand for a production model. Those numbers were set by gradient descent, pushed around on every training step so that the model's predictions improve. The signal that shapes them is the prediction loss on the surrounding tokens. A word's row therefore ends up encoding the company it keeps: which tokens come before and after it, what it predicts, what predicts it.

Nothing in that signal mentions spelling. Two words that appear in the same sentences get similar rows even if they share no letters; two words that share every letter but one can be far apart. You can see the kind of information that is there by asking which rows are closest to a word's row, measured by the cosine of the angle between them (Angle, cosine similarity and projection).

Worked example

A word's neighbors

For two words, the four vocabulary entries whose embedding rows point in the most similar direction.

import re
import numpy as np
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
E = model.tok.weight.detach().numpy()
words = [w for w in tok.vocab if re.fullmatch(r"[a-z]+", w)]

def neighbors(word, k=4):
    v = E[tok.token_id(word)]
    sims = []
    for other in words:
        if other == word:
            continue
        u = E[tok.token_id(other)]
        sims.append((float(v @ u / (np.linalg.norm(v) * np.linalg.norm(u))), other))
    return ", ".join(f"{w} {s:.2f}" for s, w in sorted(sims, reverse=True)[:k])

for word in ["tent", "weekends"]:
    print(f"{word:<9} {neighbors(word)}")
In the lesson you can change this and run it.

"tent" sits next to "pitch", "roomy", "packable" and "breathable": the words that describe tents in the catalog ("quick to pitch", "light and roomy"). "weekends" sits next to "coffee" and "nights", from phrases like "slow weekends" and "cold nights in camp". The rows encode usage. Nothing here says that "tent" has four letters or that "weekends" ends in s.

Try it now

The widget's split is approximate, like a real tokenizer's: the model gets pieces, never letters.

In your own words
In your own words: what decides the numbers in a token's embedding row, and why would two words that are spelled alike not end up near each other?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: probe the embedding for letters

Purpose. Fit two probes on the embedding rows and compare each with the guess that needs no model.

If letter information were in the rows anyway, as a side effect, you could read it out: fit a small linear model from a row's 32 numbers to the word's letter count, on part of the vocabulary, and test it on the rest. This is a probe, the same tool the interpretability lesson used to ask what a hidden state contains.

The baseline matters as much as the probe. A probe that cannot beat "guess the average length for every word" has found nothing. The next cell fits two probes with least squares: one for the letter count, one for whether the word contains the letter e.

Worked example

Two probes, two baselines

136 lowercase words, 95 for fitting and 41 held out. Least squares from the embedding row to the letter count, and to a yes/no for "contains e".

import re
import numpy as np
from tinygpt import load_pretrained

model, tok = load_pretrained("tiny-gpt")
E = model.tok.weight.detach().numpy()
words = [w for w in tok.vocab if re.fullmatch(r"[a-z]+", w)]
rows = np.array([E[tok.token_id(w)] for w in words])
X = np.concatenate([rows, np.ones((len(words), 1))], axis=1)      # a bias column
order = np.random.default_rng(0).permutation(len(words))
train, test = order[:95], order[95:]

length = np.array([float(len(w)) for w in words])
coef = np.linalg.lstsq(X[train], length[train], rcond=None)[0]
probe_error = np.abs(X[test] @ coef - length[test]).mean()
guess_error = np.abs(length[train].mean() - length[test]).mean()
print(f"letter count: probe off by {probe_error:.2f} letters on average, guessing the "
      f"mean off by {guess_error:.2f}")

has_e = np.array([1.0 if "e" in w else 0.0 for w in words])
coef = np.linalg.lstsq(X[train], has_e[train], rcond=None)[0]
probe_acc = np.mean((X[test] @ coef > 0.5) == (has_e[test] > 0.5))
majority = max(has_e[train].mean(), 1 - has_e[train].mean())
print(f"contains an e: probe right {probe_acc:.0%} of the time, always guessing the "
      f"majority {majority:.0%}")

import matplotlib.pyplot as plt

fig, (left, right) = plt.subplots(1, 2, figsize=(8, 3))
left.bar(["probe", "guess the mean"], [probe_error, guess_error])
left.set_title("Letter count: error in letters")
right.bar(["probe", "guess the majority"], [probe_acc, majority])
right.set_title("Contains e: share right")
plt.show()
In the lesson you can change this and run it.

Both probes lose to their baselines. The letter-count probe is off by 2.16 letters on average where guessing the mean is off by 1.77; the "contains e" probe is right 44% of the time where the majority guess gets 54%. A linear readout can find nothing about spelling in these rows, because nothing about spelling was ever put there. The rows are rich in another way, as the neighbors showed.

Quick check
A probe for a word's letter count is off by 2.16 letters on held-out words, while guessing the average length is off by 1.77. What is the right reading?
  1. The baseline is wrong because the mean is not a prediction a model would make
  2. The embedding encodes letter count, but a linear probe is too simple to decode it
  3. The embedding carries no usable letter-count information; the probe fitted noise
  4. The probe proves the model counts letters internally but hides the result
Show the answer

The embedding carries no usable letter-count information; the probe fitted noise

Beating the baseline is the minimum. A probe that does worse on held-out words learned the training words' quirks and nothing general, which is what you expect when the property was never part of the training signal.

Drill

Letters from pieces

Write letters_from_pieces(pieces, table, letter). table maps each piece to a dictionary of letter counts, like {"straw": {"s": 1, "t": 1, "r": 1, "a": 1, "w": 1}}. Return the total count of letter across the pieces, treating a letter that a piece's dictionary does not list as 0.

def letters_from_pieces(pieces, table, letter):
    return 0

table = {"straw": {"s": 1, "t": 1, "r": 1, "a": 1, "w": 1},
         "berry": {"b": 1, "e": 1, "r": 2, "y": 1}}
print(letters_from_pieces(["straw", "berry"], table, "r"))
In the lesson, your code is checked as you write it.

Why big models can spell a little

Production tokenizers are not word-level. Byte-pair encoding splits rare words into a few pieces, so "strawberry" might arrive as "straw" and "berry", and a made-up word as five or six fragments. That helps a little: a model can learn facts about frequent pieces. Training text does sometimes spell things out, in acronyms, lists, dictionaries, code and children's books, and from those passages a model picks up a partial table of spelling facts per piece. Counting a letter then means recalling the right facts for each piece and adding them, the chain in the stepper, with a chance to slip at every link.

That is why the behavior is inconsistent rather than absent. Common pieces are spelled well, rare ones badly; a word split one way is counted right and the same word in a different context, split differently, is not. The tokens lesson already explained the other consequences of pieces: cost, context limits and why rare words are expensive. Here the consequence is a blind spot for anything that lives below the piece.

Quick check
A production model counts the r's in "berry" correctly but miscounts them in "strawberry". Which explanation fits the mechanism?
  1. Longer words take more time, and the model runs out before it reaches the last letters
  2. One familiar piece carries reliable spelling facts; several pieces need every fact right and added
  3. Capital letters in the training data confused it about where words begin
  4. The model knows the fruit but not the compound word, so it falls back to a guess
Show the answer

One familiar piece carries reliable spelling facts; several pieces need every fact right and added

Counting runs through pieces. One frequent piece can carry a reliable spelling fact; a word split into several pieces needs every fact to be right and the addition to be right too.

Quick check
From the tokens lesson: why does a rare or misspelled word cost more tokens than a common one?
  1. The model charges extra for words it has to look up outside its vocabulary
  2. Rare words are stored in a slower part of the vocabulary that costs more to read
  3. Its pieces were never merged into one token, so it stays as several short tokens
  4. Common words are free, so only the rare ones show up on the bill
Show the answer

Its pieces were never merged into one token, so it stays as several short tokens

BPE merges the pairs that occur most often in the training text. A rare word's pieces never earned a merge, so it stays in fragments, each a token you pay for and that the model has to interpret.

Drill

Split a word into pieces

Write pieces_for(word, vocab): split word into pieces from vocab by always taking the longest piece that matches at the current position, left to right. If no piece matches, take the single character as a piece.

def pieces_for(word, vocab):
    return [word]

vocab = {"straw", "berry", "st", "raw", "aw", "b", "e", "r", "y", "tent"}
print(pieces_for("strawberry", vocab))
In the lesson, your code is checked as you write it.
In your own words
Why can a production model count the letters in a common word more reliably than in a rare one?

Write a sentence or two, then compare it with a good answer in the lesson.

In production: what this justifies

Purpose. What to hand to code, what to change in the tokens, and what to measure.

Hand character-level work to code. Counting, reversing, checking a serial number's format, comparing two spellings: a line of Python does each exactly, and a tool-using model (the tools lesson) only has to decide to call it. Asking the model to do it in its head asks for a chain of recalled spelling facts.

When you cannot use a tool, change the tokens. Spell the word out with spaces, "s t r a w b e r r y", and every letter becomes its own token, something the model can attend to directly. Models count spaced letters far better than letters hidden inside a piece. The same move helps with codes, part numbers and names.

And measure in tokens. The context limit, the bill and the speed are all per token, so a text full of rare words or numbers is longer than it looks. The tokens lesson's counting tools are the right instruments; a word count is not.

Worked example

Code counts; spacing makes letters visible

Three ways to get at the letters in a word: directly in Python, through pieces, and by spacing the word so each letter stands alone. The pieces come from the greedy splitter above.

def pieces_for(word, vocab):
    pieces, i = [], 0
    while i < len(word):
        best = word[i]
        for j in range(len(word), i, -1):
            if word[i:j] in vocab:
                best = word[i:j]
                break
        pieces.append(best)
        i += len(best)
    return pieces

vocab = {"straw", "berry", "st", "raw", "aw"}
word = "strawberry"
print("python:", word.count("r"), "r's in", len(word), "letters")
print("as pieces:", pieces_for(word, vocab), "-> the model sees",
      len(pieces_for(word, vocab)), "tokens and no letters")
spaced = " ".join(word)
print("spaced:", spaced, "->", len(spaced.split()), "tokens, one per letter;",
      spaced.split().count("r"), "of them are r")
In the lesson you can change this and run it.

Python answers in one call. As pieces, the word is two tokens with the letters sealed inside them. Spaced out, it is ten tokens, each a single letter, and a model can attend to each one; that is why the spelling trick works and why serial numbers read better with separators. The tool and the spacing both do the same thing: they move the letters from inside a token to where the model can see them.

Quick check
What does this print?
word = "strawberry"
print(word.count("r"), " ".join(word).split().count("r"), len(" ".join(word).split()))
  1. 2 3 10
  2. 3 10 10
  3. 3 3 2
  4. 3 3 10
Show the answer

3 3 10

Counting the letter directly gives 3; spacing the word out and counting tokens equal to "r" also gives 3, out of 10 single-letter tokens. Code sees letters; a model sees whatever tokens you give it.

Where this shows up in an engineer's week

A bug report says the model mangles serial numbers. Tokenize a few real ones and look at the pieces. If a code is split into odd fragments, validate it with a regular expression or a checksum in code, and show the model only what it needs to decide. Measure prompts in tokens, since cost and context limits are per token.

Goes deeper in

In your own words
A product team wants the model to validate serial numbers like "KT-2291-B". What would you build, and why?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 4 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Which statement about a token's embedding row is right?

  1. It is learned from the contexts the token appears in, so it encodes usage and nothing about spelling
  2. It is computed from the token's letters, so similar spellings give similar rows
  3. It is a hash of the token's characters
  4. It lists the token's letters as numbers

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 6 · about 35 hours of study

Modern LLM engineering

Post-training, fine-tuning, serving and safety 48 lessons 2 free
  1. Base models vs assistants FreeWhat post-training adds to a pretrained model.30 min
  2. Supervised fine-tuningTeach behavior from example conversations.40 min
  3. Instruction and preference dataThe datasets that shape assistant behavior.30 min
  4. RLHFReward models and reinforcement learning from human feedback.40 min
  5. DPO and direct preference methodsPreference training without a separate reward model.40 min
  6. Constitutional AITraining with AI feedback guided by written principles.35 min
  7. Training reasoning modelsReinforcement learning on verifiable problems.40 min
  8. Prompting vs RAG vs fine-tuningPick the cheapest approach that works.30 min
  9. LoRAFine-tune a big model by training small low-rank updates.40 min
  10. Fine-tune an open modelRun a real fine-tune and evaluate the result.45 min
  11. DistillationTrain a small model to imitate a large one.35 min
  12. Tuning an embedding model for retrievalWhen search fails on your own vocabulary: train an adapter with a contrastive loss on question and passage pairs, test on held-out passages, use batch size and hard negatives, and measure what tuning costs elsewhere.45 min
  13. QuantizationStore weights in fewer bits to run models cheaply.35 min
  14. Quantization in practice: zero points, calibration and formatsScale and zero point, per-channel scales, calibration sets, error on a layer's output, quantization-aware training and the file formats you meet at work.35 min
  15. Serving modelsBatching, throughput and latency in production.35 min
  16. One request through a serving stack: latency and costFollow one request through queue, prefill, a prefix cache, decode and the stream, and price a token from a GPU's hourly cost.35 min
  17. Speculative decodingDraft with a small model, verify with a big one.35 min
  18. Running open models locallyDownload, quantize and run a model on your own machine.30 min
  19. Will it fit? Memory for serving and fine-tuningWeights, KV cache, runtime and adapter states added up for a model card and a GPU, with a calculator.30 min
  20. InterpretabilityLooking inside models to see what they compute.40 min
  21. Confident and wrongWhy a model answers with the same assurance whether it is right, wrong or asked about something that does not exist.30 min
  22. Why it can't spell or count lettersThe model sees token IDs, and an embedding learned from usage carries nothing about spelling.25 min
  23. Copying from the promptIn-context learning as a two-head copying circuit: the base tiny GPT cannot copy a repeated pattern, a model trained on repeats grows an induction head and can.30 min
  24. Why temperature changes the answerThe same prompt at 0, 0.7 and 1.5: deterministic loops, plausible variety, broken grammar, measured on a real model.25 min
  25. Why long prompts diluteAttention is a budget shared by every token, so a fact far from the question loses its share; measured, with the cliff beyond the trained window.25 min
  26. Why it never says "I don't know"A chat model answers every request because its fine-tuning data always did; preference training adds flattery; how to build abstention in.25 min
  27. Benchmarks and their limitsWhat leaderboards measure, and what they miss.38 min
  28. Alignment and safetyMaking capable models behave as intended.35 min
  29. Evaluating agentsMeasure multi-step, tool-using behavior.35 min
  30. Reading AI papersGet the idea out of a research paper in 30 minutes.30 min
  31. Capstone: a tuned, evaluated assistantFine-tune, evaluate and deploy a specialist assistant.45 min
  32. Weight-space programming: what it is, and what has been shown Free OptionalA language model reads and writes another network's weights with no training loop: the published experiments, their limits and the criticism, and one edit you make by hand.30 min
  33. Anatomy of a checkpoint OptionalA model file is a set of named tensors: count them, find the ones that write to the residual stream, and save one to bytes and back.35 min
  34. Reading weights without running the model OptionalSingular values, and an attention head's token-to-token circuit, read from the matrices alone and then checked against a measurement.35 min
  35. Writing weights by hand OptionalSet a small network's 55 weights so it catches every ball in a paddle game, the way a program would be written, and meet compiled transformers.40 min
  36. Edits are deltas: task vectors, patches and hashes OptionalSubtract a base from a fine-tune and the edit becomes an object: scale it, take it away, ship it as a patch, and prove by hash what was applied.35 min
  37. Merging models OptionalAverage two fine-tunes, add their task vectors, try SLERP, TIES and DARE, measure the interference, and compare with a model trained on both tasks.40 min
  38. Low-rank edits OptionalCut a fine-tune's change down to its largest directions with the SVD: how few a behavior needs, and how to pull an adapter out of a delta.35 min
  39. Directions: steer the run or edit the weights OptionalFind a behavior's direction in the residual stream, add it at run time, project it out of the weights, and see the effect change from input to input.40 min
  40. Editing one fact OptionalA rank-one change to one matrix rewrites what a product is made from; then reliability, generalization, locality, and what breaks after many edits.40 min
  41. The editor's harness OptionalWhat an LLM that edits weights is given and what it is denied: weights as text and their cost in tokens, a sandbox, a budget of forward passes, and rules the harness enforces.40 min
  42. An LLM reads a network's weights OptionalSix recorded sessions: an LLM is handed 1,315 weights as text and asked what the network was trained to do. Score each answer against the network, and study one it got wrong.40 min
  43. An LLM fixes a model's weights OptionalFind a tiny GPT's loop by hand, then replay recorded sessions in which an LLM hunts for it with ten forward passes, and measure every fix on held-out questions.45 min
  44. An LLM writes a network's weights OptionalFive recorded sessions: an LLM writes every weight of a paddle network from the game's source and one sentence. Score each against its specification, test its design claims, and edit one by hand.40 min
  45. Did it work? Testing an edit OptionalReview a patch for a tiny GPT that can't stop talking: held-out questions, a paired comparison, a same-size control, and what the edit cost.40 min
  46. The published edit, on a real model OptionalRead the edit published for an open model of 870 million weights, apply and verify one in the same form, see what 16-bit and 4-bit storage do to a fix, and run the real thing from a notebook.45 min
  47. Safety, licenses and provenance OptionalBefore a custom model is released: trace its lineage by hash, work out what its parents' licenses pass on, see what an edit can do to a safeguard, and write its card.40 min
  48. Capstone: ship a custom model OptionalSet release gates, then diagnose, edit, measure and document a tiny GPT with a fault no lesson has fixed, and write the record a stranger could check.45 min
Part of a free course 6 lessons

Why Models Do That: six behaviors, traced to the mechanism

Confident wrong answers, miscounted letters, copying from the prompt, temperature, long prompts that dilute, and a model that never says "I don't know": each behavior reproduced on a tiny GPT with every number printed by code, then the mechanism behind it and the Phase 1 technique it justifies.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write