Build an LLM from scratch: a free course
Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.
Match your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.
The phase has built a GPT by hand and trained one in the page, but the model people are paid to use lives in a library. This lesson is the bridge. GPT-2 is the same design as the model you wrote, stored under different names, with one layout difference and a tied head. Once you can translate the names and shapes yourself, a library call stops being magic: from_pretrained is a loop over a table you can write.
You will map every node of the model to a GPT-2 tensor, count its parameters from the pieces, load renamed weights into your own class and check the logits match, then pad a batch the way the library does and read the config and the generate arguments.
.T. A guess: what shape does this print?import torch w = torch.zeros(768, 2304) print(tuple(w.T.shape))
(2304, 768)
.T swaps the two dimensions, so (768, 2304) becomes (2304, 768). The same flip is the whole difference between how GPT-2 and this course store a linear layer's weight.
This rests on The GPT architecture, which counted GPT-2's 124,439,808 parameters, and on Build a tiny GPT, where you wrote the same design as a PyTorch class. GPT-2 is the model you built, with more layers, more width and a vocabulary of 50,257, and its weights are a public file. Loading them is mostly a matter of names and shapes.
Every tensor in a PyTorch model has a name in its state_dict. Your GPT calls the token table tok.weight. GPT-2 calls it wte.weight, for word token embedding, and prefixes every name with transformer. when the model also has a language-model head. The table below is the whole translation for one block.
NAMES = {"qkv": "attn.c_attn", "proj": "attn.c_proj"}
print(NAMES["qkv"])attn.c_attn
The dictionary looks up the course's name qkv and returns GPT-2's name for the same layer, attn.c_attn. Every renaming in this lesson is a lookup like this one.
| Your class | GPT-2 state_dict | Shape in GPT-2 (768 wide) |
|---|---|---|
tok.weight | wte.weight | (50257, 768) |
pos.weight | wpe.weight | (1024, 768) |
blocks.0.ln1 | h.0.ln_1 | (768,) and (768,) |
blocks.0.qkv | h.0.attn.c_attn | weight (768, 2304), bias (2304,) |
blocks.0.proj | h.0.attn.c_proj | weight (768, 768), bias (768,) |
blocks.0.ln2 | h.0.ln_2 | (768,) and (768,) |
blocks.0.mlp.0 | h.0.mlp.c_fc | weight (768, 3072), bias (3072,) |
blocks.0.mlp.2 | h.0.mlp.c_proj | weight (3072, 768), bias (768,) |
ln_f | ln_f | (768,) and (768,) |
head | lm_head, the same tensor as wte | (50257, 768) |
Two differences matter. GPT-2 stores each linear layer's weight as (in, out), because it uses a layer called Conv1D; PyTorch's nn.Linear, which your class uses, stores (out, in). So every weight in the four linear rows is flipped. And GPT-2 ties its head to its token table: one tensor does both jobs, so the head has no bias and no separate parameters.
The weight of c_attn, shape (768, 2304)
Only the weights of the four linear layers in a block (c_attn, c_proj, c_fc and the MLP's c_proj) are flipped. Tables, biases and norms keep their shape.
124,439,808
38,597,376 for the token table, 786,432 for positions, 12 blocks of 7,087,872 and a final norm of 1,536 add up to 124,439,808. Counting the head separately would add another 38,597,376.
Write a sentence or two, then compare it with a good answer in the lesson.
The translation is a short function in each direction. The cell builds a small GPT with this course's class, copies it into GPT-2's names and shapes, loads that into a second instance through the reverse function, and checks that both give the same logits. Real GPT-2 weights go through the same functions in the notebook.
A 2-layer GPT with d = 8. The head is tied to the token table first, because GPT-2 has no separate head.
import torch from tinygpt import GPT NAMES = {"ln1": "ln_1", "ln2": "ln_2", "qkv": "attn.c_attn", "proj": "attn.c_proj", "mlp.0": "mlp.c_fc", "mlp.2": "mlp.c_proj"} LINEAR = ("qkv", "proj", "mlp.0", "mlp.2") # GPT-2 stores these weights as (in, out) def to_gpt2(sd): out = {"wte.weight": sd["tok.weight"], "wpe.weight": sd["pos.weight"], "ln_f.weight": sd["ln_f.weight"], "ln_f.bias": sd["ln_f.bias"]} for name, t in sd.items(): if name.startswith("blocks."): _, i, rest = name.split(".", 2) layer, kind = rest.rsplit(".", 1) flip = kind == "weight" and layer in LINEAR out[f"h.{i}.{NAMES[layer]}.{kind}"] = t.T if flip else t return out def from_gpt2(sd, V): back = {v: k for k, v in NAMES.items()} out = {"tok.weight": sd["wte.weight"], "pos.weight": sd["wpe.weight"], "ln_f.weight": sd["ln_f.weight"], "ln_f.bias": sd["ln_f.bias"], "head.weight": sd["wte.weight"], "head.bias": torch.zeros(V)} for name, t in sd.items(): if name.startswith("h."): _, i, rest = name.split(".", 2) layer, kind = rest.rsplit(".", 1) layer = back[layer] flip = kind == "weight" and layer in LINEAR out[f"blocks.{i}.{layer}.{kind}"] = t.T if flip else t return out torch.manual_seed(0) V, ctx, d, layers, heads = 50, 16, 8, 2, 2 mine = GPT(V, ctx, d, layers, heads) with torch.no_grad(): # tie the head, as GPT-2 does mine.head.weight.copy_(mine.tok.weight) mine.head.bias.zero_() g = to_gpt2(mine.state_dict()) for name in ("wte.weight", "h.0.attn.c_attn.weight", "h.0.attn.c_proj.weight"): print(name, tuple(g[name].shape)) other = GPT(V, ctx, d, layers, heads) other.load_state_dict(from_gpt2(g, V)) ids = torch.tensor([[3, 14, 15, 9, 2]]) print("same logits:", torch.allclose(mine(ids), other(ids)), " tensors:", len(g))
The token table keeps (50, 8), but c_attn becomes (8, 24): the flip. The square c_proj stays (8, 8) either way, which is the dangerous one, because a missed flip there would load without any error. After the round trip the second model gives the same logits as the first, and the GPT-2 view holds 28 tensors: 2 embeddings, the final norm's 2, and 12 for each of the 2 blocks.
nn.Linear without flipping it. What happens?No error, and the model gives wrong answers
A square matrix has the same shape either way, so the load succeeds. But a weight is not symmetric, so the layer computes with the transposed matrix and the outputs are wrong. Check outputs against the original, not just that loading did not fail.
Write gpt2_name(name) for names like tok.weight, ln_f.bias and blocks.3.qkv.weight. Use the table: tok is wte, pos is wpe, ln_f stays, and inside a block ln1, ln2, qkv, proj, mlp.0 and mlp.2 become ln_1, ln_2, attn.c_attn, attn.c_proj, mlp.c_fc and mlp.c_proj under h.<number>..
NAMES = {"ln1": "ln_1", "ln2": "ln_2", "qkv": "attn.c_attn", "proj": "attn.c_proj",
"mlp.0": "mlp.c_fc", "mlp.2": "mlp.c_proj"}
def gpt2_name(name):
return name
print(gpt2_name("blocks.3.qkv.weight"))
ids = [[5, 6, 7], [8]] T = max(len(x) for x in ids) print([[0] * (T - len(x)) + [1] * len(x) for x in ids])
[[1, 1, 1], [0, 0, 1]]
The first row has three real tokens, all 1s. The second has two pads on the left, 0s, and its one real token, a 1. The mask follows the padding.
A model takes a rectangle of ids, so shorter prompts are padded. For generation the padding goes on the left, so every row ends at the last column, where the next token is predicted. An attention mask of 1s for real tokens and 0s for padding tells attention to ignore the padding: the scores at padded keys are set to minus infinity before the softmax, so they get weight 0.
The same two prompts as the real tokenizer's output below: GPT-2's end-of-text id, 50,256, is the pad.
import torch ids = [[464, 3797, 3332], [15496]] # "The cat sat" and "Hello" PAD = 50256 T = max(len(x) for x in ids) batch = torch.tensor([[PAD] * (T - len(x)) + x for x in ids]) mask = torch.tensor([[0] * (T - len(x)) + [1] * len(x) for x in ids]) print(batch.tolist()) print(mask.tolist()) scores = torch.tensor([[0.5, 1.0, 2.0], [0.3, 0.1, 2.0]]) # last token's score per key w = torch.softmax(scores.masked_fill(mask == 0, float("-inf")), dim=-1) print([[round(v, 2) for v in row] for row in w.tolist()])
The second prompt is one token, so two pads go on its left. Its mask row is [0, 0, 1], and the softmax gives the padded keys exactly 0 and the real key all of the weight: [0.0, 0.0, 1.0]. The first row has no padding and keeps its three weights, 0.14, 0.23 and 0.63.
Write left_pad(seqs, pad) that returns (batch, mask): every sequence padded on the left with pad to the length of the longest, and a mask with 0 for each pad and 1 for each real token.
def left_pad(seqs, pad): return seqs, seqs print(left_pad([[5, 6, 7], [8]], 0))
Write masked_weights(scores, mask): scores are a (B, T) tensor of attention scores for one query, and mask is a (B, T) tensor of 1s and 0s. Return the softmax over the last dimension after setting the scores where the mask is 0 to minus infinity.
import torch def masked_weights(scores, mask): return torch.softmax(scores, dim=-1) print(masked_weights(torch.tensor([[1.0, 2.0, 3.0]]), torch.tensor([[0, 1, 1]])))
Write a sentence or two, then compare it with a good answer in the lesson.
In Hugging Face, AutoModelForCausalLM.from_pretrained("gpt2") downloads the tensors you just renamed, builds the class, and loads them. Everything it does is what the cells above did. Two things are worth reading before you rely on it: the tokenizer's batch output, and the config.json that fixes the sizes.
64
The width 768 is split over 12 heads: 768 / 12 = 64 numbers per head.
On the job: loading a checkpoint is where silent bugs live. After any conversion, compare the logits of your model and the reference on one prompt and require them to match to about 1e-5; shape checks pass even when a square weight is flipped wrong. Pin the library and model versions, since a renamed key or a changed default can alter outputs with no error. And read the
config.jsonbefore you read the model card: it tells you the layers, the heads, the context and whether the head is tied.
AutoTokenizer loads.Write a sentence or two, then compare it with a good answer in the lesson.
Why can you not load GPT-2's c_attn weight straight into a PyTorch nn.Linear of the same sizes?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write