Index Ai Academy
Phase 5 · The transformer

From your GPT to GPT-2 in Hugging Face

Match your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.

About 40 minutes · Versions checked November 2026 · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers8

Why it matters

The phase has built a GPT by hand and trained one in the page, but the model people are paid to use lives in a library. This lesson is the bridge. GPT-2 is the same design as the model you wrote, stored under different names, with one layout difference and a tied head. Once you can translate the names and shapes yourself, a library call stops being magic: from_pretrained is a loop over a table you can write.

You will map every node of the model to a GPT-2 tensor, count its parameters from the pieces, load renamed weights into your own class and check the logits match, then pad a batch the way the library does and read the config and the generate arguments.

What you'll be able to do

  • Match each tensor of this course's GPT to its name and shape in GPT-2's state_dict, and say why the linear weights are flipped
  • Count GPT-2 small's 124,439,808 parameters from its pieces, with the head tied
  • Pad a batch on the left and build the attention mask the library expects
  • Read a config.json and the arguments of generate, and check a converted model against the original
Quick check
A weight matrix with 768 rows and 2,304 columns is flipped with .T. A guess: what shape does this print?
import torch

w = torch.zeros(768, 2304)
print(tuple(w.T.shape))
  1. (768,)
  2. (2304, 768)
  3. (1769472,)
  4. (768, 2304)
Show the answer

(2304, 768)

.T swaps the two dimensions, so (768, 2304) becomes (2304, 768). The same flip is the whole difference between how GPT-2 and this course store a linear layer's weight.

The idea: the same GPT under different names

This rests on The GPT architecture, which counted GPT-2's 124,439,808 parameters, and on Build a tiny GPT, where you wrote the same design as a PyTorch class. GPT-2 is the model you built, with more layers, more width and a vocabulary of 50,257, and its weights are a public file. Loading them is mostly a matter of names and shapes.

Every tensor in a PyTorch model has a name in its state_dict. Your GPT calls the token table tok.weight. GPT-2 calls it wte.weight, for word token embedding, and prefixes every name with transformer. when the model also has a language-model head. The table below is the whole translation for one block.

Quick check
A dictionary maps this course's layer names to GPT-2's. A guess: what does this print?
NAMES = {"qkv": "attn.c_attn", "proj": "attn.c_proj"}
print(NAMES["qkv"])
  1. attn.c_attn
  2. qkv
  3. attn.qkv
  4. c_attn
Show the answer

attn.c_attn

The dictionary looks up the course's name qkv and returns GPT-2's name for the same layer, attn.c_attn. Every renaming in this lesson is a lookup like this one.

Your classGPT-2 state_dictShape in GPT-2 (768 wide)
tok.weightwte.weight(50257, 768)
pos.weightwpe.weight(1024, 768)
blocks.0.ln1h.0.ln_1(768,) and (768,)
blocks.0.qkvh.0.attn.c_attnweight (768, 2304), bias (2304,)
blocks.0.projh.0.attn.c_projweight (768, 768), bias (768,)
blocks.0.ln2h.0.ln_2(768,) and (768,)
blocks.0.mlp.0h.0.mlp.c_fcweight (768, 3072), bias (3072,)
blocks.0.mlp.2h.0.mlp.c_projweight (3072, 768), bias (768,)
ln_fln_f(768,) and (768,)
headlm_head, the same tensor as wte(50257, 768)

Two differences matter. GPT-2 stores each linear layer's weight as (in, out), because it uses a layer called Conv1D; PyTorch's nn.Linear, which your class uses, stores (out, in). So every weight in the four linear rows is flipped. And GPT-2 ties its head to its token table: one tensor does both jobs, so the head has no bias and no separate parameters.

Quick check
Which of these tensors does GPT-2 store flipped, as (in, out), compared with an nn.Linear layer?
  1. The bias of c_attn, shape (2304,)
  2. The layer norm weight ln_1, shape (768,)
  3. The weight of c_attn, shape (768, 2304)
  4. The token table wte, shape (50257, 768)
Show the answer

The weight of c_attn, shape (768, 2304)

Only the weights of the four linear layers in a block (c_attn, c_proj, c_fc and the MLP's c_proj) are flipped. Tables, biases and norms keep their shape.

Quick check
From the GPT architecture lesson: GPT-2 small has d = 768, 12 layers, a context of 1,024 and a vocabulary of 50,257. How many parameters does it have, with the head tied to the token table?
  1. 124,439,808
  2. 163,037,184
  3. 85,054,464
  4. 38,597,376
Show the answer

124,439,808

38,597,376 for the token table, 786,432 for positions, 12 blocks of 7,087,872 and a final norm of 1,536 add up to 124,439,808. Counting the head separately would add another 38,597,376.

Try it now
In your own words
Why does GPT-2's c_attn weight have shape (768, 2304), when your class's qkv weight is (2304, 768)?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: GPT-2's names in your own class

The translation is a short function in each direction. The cell builds a small GPT with this course's class, copies it into GPT-2's names and shapes, loads that into a second instance through the reverse function, and checks that both give the same logits. Real GPT-2 weights go through the same functions in the notebook.

Worked example

Your GPT in GPT-2's names, and back

A 2-layer GPT with d = 8. The head is tied to the token table first, because GPT-2 has no separate head.

import torch
from tinygpt import GPT

NAMES = {"ln1": "ln_1", "ln2": "ln_2", "qkv": "attn.c_attn", "proj": "attn.c_proj",
         "mlp.0": "mlp.c_fc", "mlp.2": "mlp.c_proj"}
LINEAR = ("qkv", "proj", "mlp.0", "mlp.2")     # GPT-2 stores these weights as (in, out)

def to_gpt2(sd):
    out = {"wte.weight": sd["tok.weight"], "wpe.weight": sd["pos.weight"],
           "ln_f.weight": sd["ln_f.weight"], "ln_f.bias": sd["ln_f.bias"]}
    for name, t in sd.items():
        if name.startswith("blocks."):
            _, i, rest = name.split(".", 2)
            layer, kind = rest.rsplit(".", 1)
            flip = kind == "weight" and layer in LINEAR
            out[f"h.{i}.{NAMES[layer]}.{kind}"] = t.T if flip else t
    return out

def from_gpt2(sd, V):
    back = {v: k for k, v in NAMES.items()}
    out = {"tok.weight": sd["wte.weight"], "pos.weight": sd["wpe.weight"],
           "ln_f.weight": sd["ln_f.weight"], "ln_f.bias": sd["ln_f.bias"],
           "head.weight": sd["wte.weight"], "head.bias": torch.zeros(V)}
    for name, t in sd.items():
        if name.startswith("h."):
            _, i, rest = name.split(".", 2)
            layer, kind = rest.rsplit(".", 1)
            layer = back[layer]
            flip = kind == "weight" and layer in LINEAR
            out[f"blocks.{i}.{layer}.{kind}"] = t.T if flip else t
    return out

torch.manual_seed(0)
V, ctx, d, layers, heads = 50, 16, 8, 2, 2
mine = GPT(V, ctx, d, layers, heads)
with torch.no_grad():                           # tie the head, as GPT-2 does
    mine.head.weight.copy_(mine.tok.weight)
    mine.head.bias.zero_()
g = to_gpt2(mine.state_dict())
for name in ("wte.weight", "h.0.attn.c_attn.weight", "h.0.attn.c_proj.weight"):
    print(name, tuple(g[name].shape))
other = GPT(V, ctx, d, layers, heads)
other.load_state_dict(from_gpt2(g, V))
ids = torch.tensor([[3, 14, 15, 9, 2]])
print("same logits:", torch.allclose(mine(ids), other(ids)), " tensors:", len(g))
It runs right here in your browser, with nothing to install.

The token table keeps (50, 8), but c_attn becomes (8, 24): the flip. The square c_proj stays (8, 8) either way, which is the dangerous one, because a missed flip there would load without any error. After the round trip the second model gives the same logits as the first, and the GPT-2 view holds 28 tensors: 2 embeddings, the final norm's 2, and 12 for each of the 2 blocks.

Quick check
In GPT-2 the attention output layer c_proj is square, (768, 768). Someone loads its weight into an nn.Linear without flipping it. What happens?
  1. Loading works and training repairs it within a few steps
  2. The model gives the same answers, since a square matrix is symmetric
  3. No error, and the model gives wrong answers
  4. A shape error stops the load
Show the answer

No error, and the model gives wrong answers

A square matrix has the same shape either way, so the load succeeds. But a weight is not symmetric, so the layer computes with the transposed matrix and the outputs are wrong. Check outputs against the original, not just that loading did not fail.

Drill

Rename a tensor

Write gpt2_name(name) for names like tok.weight, ln_f.bias and blocks.3.qkv.weight. Use the table: tok is wte, pos is wpe, ln_f stays, and inside a block ln1, ln2, qkv, proj, mlp.0 and mlp.2 become ln_1, ln_2, attn.c_attn, attn.c_proj, mlp.c_fc and mlp.c_proj under h.<number>..

NAMES = {"ln1": "ln_1", "ln2": "ln_2", "qkv": "attn.c_attn", "proj": "attn.c_proj",
         "mlp.0": "mlp.c_fc", "mlp.2": "mlp.c_proj"}


def gpt2_name(name):
    return name


print(gpt2_name("blocks.3.qkv.weight"))
In the lesson, your code is checked as you write it.
Quick check
Two prompts, 3 and 1 tokens, are left-padded with the id 0 and given a mask. A guess: what does this print?
ids = [[5, 6, 7], [8]]
T = max(len(x) for x in ids)
print([[0] * (T - len(x)) + [1] * len(x) for x in ids])
  1. [[0, 0, 0], [1, 1, 0]]
  2. [[1, 1, 1], [0, 0, 1]]
  3. [[1, 1, 1], [1, 0, 0]]
  4. [[1, 1, 1], [1, 1, 1]]
Show the answer

[[1, 1, 1], [0, 0, 1]]

The first row has three real tokens, all 1s. The second has two pads on the left, 0s, and its one real token, a 1. The mask follows the padding.

A batch of different lengths

A model takes a rectangle of ids, so shorter prompts are padded. For generation the padding goes on the left, so every row ends at the last column, where the next token is predicted. An attention mask of 1s for real tokens and 0s for padding tells attention to ignore the padding: the scores at padded keys are set to minus infinity before the softmax, so they get weight 0.

Worked example

Pad a batch and mask the padding

The same two prompts as the real tokenizer's output below: GPT-2's end-of-text id, 50,256, is the pad.

import torch

ids = [[464, 3797, 3332], [15496]]               # "The cat sat" and "Hello"
PAD = 50256
T = max(len(x) for x in ids)
batch = torch.tensor([[PAD] * (T - len(x)) + x for x in ids])
mask = torch.tensor([[0] * (T - len(x)) + [1] * len(x) for x in ids])
print(batch.tolist())
print(mask.tolist())

scores = torch.tensor([[0.5, 1.0, 2.0], [0.3, 0.1, 2.0]])   # last token's score per key
w = torch.softmax(scores.masked_fill(mask == 0, float("-inf")), dim=-1)
print([[round(v, 2) for v in row] for row in w.tolist()])
In the lesson you can change this and run it.

The second prompt is one token, so two pads go on its left. Its mask row is [0, 0, 1], and the softmax gives the padded keys exactly 0 and the real key all of the weight: [0.0, 0.0, 1.0]. The first row has no padding and keeps its three weights, 0.14, 0.23 and 0.63.

Drill

Pad a batch on the left

Write left_pad(seqs, pad) that returns (batch, mask): every sequence padded on the left with pad to the length of the longest, and a mask with 0 for each pad and 1 for each real token.

def left_pad(seqs, pad):
    return seqs, seqs


print(left_pad([[5, 6, 7], [8]], 0))
In the lesson, your code is checked as you write it.
Drill

Weights that ignore the padding

Write masked_weights(scores, mask): scores are a (B, T) tensor of attention scores for one query, and mask is a (B, T) tensor of 1s and 0s. Return the softmax over the last dimension after setting the scores where the mask is 0 to minus infinity.

import torch


def masked_weights(scores, mask):
    return torch.softmax(scores, dim=-1)


print(masked_weights(torch.tensor([[1.0, 2.0, 3.0]]), torch.tensor([[0, 1, 1]])))
In the lesson, your code is checked as you write it.
In your own words
Why is padding put on the left for generation, and what does the attention mask do?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: the library call, the config and the arguments

In Hugging Face, AutoModelForCausalLM.from_pretrained("gpt2") downloads the tensors you just renamed, builds the class, and loads them. Everything it does is what the cells above did. Two things are worth reading before you rely on it: the tokenizer's batch output, and the config.json that fixes the sizes.

Quick check
A config.json says n_layer 12, n_head 12, n_embd 768. How many numbers does each head work with in its slice of q, k and v?
  1. 64
  2. 12
  3. 9216
  4. 768
Show the answer

64

The width 768 is split over 12 heads: 768 / 12 = 64 numbers per head.

Where this shows up in an engineer's week

On the job: loading a checkpoint is where silent bugs live. After any conversion, compare the logits of your model and the reference on one prompt and require them to match to about 1e-5; shape checks pass even when a square weight is flipped wrong. Pin the library and model versions, since a renamed key or a changed default can alter outputs with no error. And read the config.json before you read the model card: it tells you the layers, the heads, the context and whether the head is tied.

Goes deeper in

In your own words
A converted model loads with no error but writes nonsense. What do you check first?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 2 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

Why can you not load GPT-2's c_attn weight straight into a PyTorch nn.Linear of the same sizes?

  1. GPT-2 stores (in, out) where nn.Linear stores (out, in)
  2. GPT-2's weights use a different number format
  3. The names differ, so PyTorch cannot find it
  4. GPT-2 has no q, k and v, only one score

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 22 lessons

Build an LLM from scratch: a free course

Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write