Index Ai Academy
Phase 5 · The transformer

Training a GPT well

Warmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers10

Why it matters

The tiny GPT you built trained in 300 steps with a fixed learning rate. A real run is thousands of steps, and four habits keep it healthy: a warmup, a decaying rate, a clip on the gradient and a validation loss that you can trust. They are small changes to the loop you already wrote, and each has a number you can read in the log.

This lesson recalls the schedule and the training script from Phase 4 and teaches only what is particular to a GPT. You will write the schedule and the clip, train the model with them, watch a sweep of learning rates, and read the curves.

What you'll be able to do

  • Write the warmup-then-cosine schedule and the gradient-clipping factor, and compute both by hand
  • Train the course's GPT with a LambdaLR schedule, clip_grad_norm_ and a validation loss averaged over many batches
  • Read a loss curve and a gradient norm to tell a learning rate that is too high from one that is too low
  • Name what to log, and what to save in a checkpoint, for a run that must be resumed
Quick check
Warmup ramps the learning rate up over the first steps: step 4 of a 10-step warmup to a peak of 0.003 uses a fifth of the way up. A guess: what does this print?
print(3e-3 * (4 + 1) / 10)
  1. 0.0003
  2. 0.003
  3. 0.0015
  4. 0.0005
Show the answer

0.0015

The warmup rule is peak x (step + 1) / warmup, so step 4 gives 0.003 x 5 / 10 = 0.0015: halfway up, because steps are counted from 0.

The idea: four habits that make a GPT train well

This rests on Learning-rate schedules in PyTorch, which built warmup and cosine decay by hand, and on A training script, start to finish, where a training run had a config, a seed and a checkpoint. Here is only what is particular to a GPT, on the model you built in Build a tiny GPT.

A 300-step demo with a fixed learning rate trains a GPT, but a real run is long, and four things keep it healthy.

  1. Warmup. The optimizer's running estimates of each gradient's size start from nothing. In the first steps they are poor, so a full-size learning rate can throw the weights somewhere bad. The rate climbs from near zero to its peak over the first steps.
  2. Cosine decay. After the peak the rate falls along half a cosine to a floor, here 10% of the peak, so the late steps make small, careful changes.
  3. Gradient clipping. A rare batch can produce a gradient many times the usual size. Clipping rescales the whole gradient so its length is never above a limit, and the direction stays the same.
  4. A trustworthy validation loss. One batch of validation text gives a noisy number. Averaging over many batches, with no gradients and the model in evaluation mode, gives one you can compare between runs.
Worked example

The schedule, printed

The setup's lr_at is warmup then cosine to a floor. Ten steps of warmup, 100 in all, peak 0.003.

Setup (runs before every cell) · 57 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import math
import re
import requests
import torch
import torch.nn as nn
from tinygpt import GPT

text = requests.get("https://api.gradient.test/v1/corpus").json()["text"]
tokens = re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", text)
vocab = sorted(set(tokens))
stoi = {t: i for i, t in enumerate(vocab)}
data = torch.tensor([stoi[t] for t in tokens])
split = int(0.9 * len(data))
train_data, val_data = data[:split], data[split:]

def get_batch(d, block=24, batch=16):
    starts = torch.randint(0, len(d) - block - 1, (batch,)).tolist()
    x = torch.stack([d[i:i + block] for i in starts])
    y = torch.stack([d[i + 1:i + block + 1] for i in starts])
    return x, y

def loss_of(model, x, y):
    return nn.functional.cross_entropy(model(x).view(-1, len(vocab)), y.view(-1))

def lr_at(step, peak, warmup, total):
    if step < warmup:
        return peak * (step + 1) / warmup
    progress = (step - warmup) / (total - warmup)
    return 0.1 * peak + 0.9 * peak * 0.5 * (1 + math.cos(math.pi * progress))

@torch.no_grad()
def estimate_loss(model, d, iters=10, batch=32):
    model.eval()
    total = sum(loss_of(model, *get_batch(d, batch=batch)).item() for _ in range(iters))
    model.train()
    return total / iters

def run(peak, steps=100, warmup=10, clip=1.0, every=20, seed=0):
    torch.manual_seed(seed)
    model = GPT(len(vocab), 24, 32, 2, 4)
    opt = torch.optim.AdamW(model.parameters(), lr=peak, weight_decay=0.01)
    sched = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: lr_at(s, peak, warmup, steps) / peak)
    train, val, biggest = [], [], 0.0
    for step in range(steps):
        x, y = get_batch(train_data)
        loss = loss_of(model, x, y)
        opt.zero_grad()
        loss.backward()
        norm = nn.utils.clip_grad_norm_(model.parameters(), clip)
        biggest = max(biggest, float(norm))
        opt.step()
        sched.step()
        train.append(loss.item())
        if step % every == 0 or step == steps - 1:
            val.append((step, estimate_loss(model, val_data)))
    return model, train, val, biggest
print([round(lr_at(s, 3e-3, 10, 100), 6) for s in range(0, 100, 10)])
print(round(lr_at(99, 3e-3, 10, 100), 6))
In the lesson you can change this and run it.

The rate starts at 0.0003, a tenth of the peak, reaches 0.003 at step 10, and then falls along the cosine: 0.001884 at step 50, 0.000381 at step 90 and 0.000301 at the last step, a little above the floor of 0.0003.

Try it now

The values printed above, every 10 steps: a short climb, then the cosine down toward a tenth of the peak.

Quick check
From the learning-rate schedules lesson: what is warmup for?
  1. It lets the model see the validation data before training
  2. It makes the model train for a larger number of steps
  3. It keeps the first noisy updates small while estimates settle
  4. It raises the final loss on purpose to avoid overfitting
Show the answer

It keeps the first noisy updates small while estimates settle

In the first steps the optimizer's gradient statistics are unreliable, so a small learning rate keeps early updates safe. The rate then climbs to its peak.

Quick check
A batch's gradient has length 5.0 and the limit is 1.0. By what factor does clipping scale every number in it?
  1. 5
  2. 1
  3. 0.2
  4. 0.5
Show the answer

0.2

Clipping multiplies the gradient by limit / length = 1 / 5 = 0.2, so the new length is exactly 1.0 and the direction is unchanged.

In your own words
Why does a long GPT run need warmup, decay and clipping, when a 300-step demo runs fine without them?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: the schedule, the clip and the check in a loop

The lesson's setup, folded at the first cell, loads the practice corpus, cuts it into windows, and defines lr_at, estimate_loss and run, the training loop. The three lines that matter are these:

fn = lambda s: lr_at(s, peak, warmup, steps) / peak
sched = torch.optim.lr_scheduler.LambdaLR(opt, fn)
norm = nn.utils.clip_grad_norm_(model.parameters(), clip)   # between backward and step
sched.step()                                                # after opt.step()

LambdaLR multiplies the optimizer's base rate by a function of the step, which is why the function returns a fraction of the peak. clip_grad_norm_ rescales every gradient in place and returns the length it measured before. The loop also estimates the validation loss every 20 steps.

Worked example

One run with warmup, decay and clipping

A 2-layer GPT, 100 steps, a peak rate of 0.03, which this small model likes. Training loss every step, validation loss estimated over 10 batches every 20 steps.

Uses the setup above, 57 lines.

model, train, val, biggest = run(3e-2)
for step, v in val:
    print(f"step {step:>3}  validation loss {v:.3f}")
print("final training loss", round(train[-1], 3))
print("largest gradient norm", round(biggest, 2))

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.5, 3))
ax.plot(train, label="training (every step)")
steps = [s for s, v in val]
ax.plot(steps, [v for s, v in val], "o-", label="validation (10 batches)")
ax.set_xlabel("step")
ax.set_ylabel("loss")
ax.legend()
In the lesson you can change this and run it.

The validation loss falls from 5.336 at step 0 to about 1.0 by the end, and the training loss ends near 0.95. The two curves track each other, which says the model is not memorizing yet. The largest gradient norm was 0.73, under the limit of 1.0, so clipping never fired on this run: it is insurance, and the next experiment shows it working.

Quick check
LambdaLR multiplies the optimizer's base rate by a function of the step. With base rate 0.03 and the schedule above, what factor does the function return at step 0? A guess: what does this print?
import math

def lr_at(step, peak, warmup, total):
    if step < warmup:
        return peak * (step + 1) / warmup
    return peak

print(round(lr_at(0, 3e-2, 10, 100) / 3e-2, 2))
  1. 0.03
  2. 1.0
  3. 0.1
  4. 0.0
Show the answer

0.1

At step 0 the schedule gives 0.03 x 1 / 10 = 0.003, which is 0.1 of the base rate, so the factor is 0.1.

Drill

Write the schedule

Write lr_schedule(step, peak, warmup, total): during the first warmup steps the rate is peak * (step + 1) / warmup. After that it falls along half a cosine from peak at step warmup to 0.1 * peak at step total: 0.1 * peak + 0.9 * peak * 0.5 * (1 + cos(pi * progress)), where progress is (step - warmup) / (total - warmup).

Uses the setup above, 57 lines.

import math


def lr_schedule(step, peak, warmup, total):
    return peak


print(round(lr_schedule(4, 3e-3, 10, 100), 6))
In the lesson, your code is checked as you write it.
Drill

Clip a gradient

Write clip_scale(norm, max_norm): the factor to multiply a gradient by so that its length is at most max_norm. If the gradient is already short enough, the factor is 1.0; otherwise it is max_norm / norm.

Uses the setup above, 57 lines.

def clip_scale(norm, max_norm):
    return 1.0


print(clip_scale(5.0, 1.0))
In the lesson, your code is checked as you write it.
Worked example

One batch against ten

The same kind of trained model, scored eight times with one batch and eight times with ten. The model is fixed; only the estimate changes.

Uses the setup above, 57 lines.

model, train, val, biggest = run(3e-2)
torch.manual_seed(5)
one = [estimate_loss(model, val_data, iters=1) for _ in range(8)]
ten = [estimate_loss(model, val_data, iters=10) for _ in range(8)]
print("one batch:  ", round(min(one), 3), "to", round(max(one), 3))
print("ten batches:", round(min(ten), 3), "to", round(max(ten), 3))
In the lesson you can change this and run it.

One batch of 32 windows swings between 0.955 and 1.081 for the same model, a gap of 0.126. Ten batches narrow it to 0.990 to 1.023, a gap of 0.033. A change of 0.05 between two runs would be noise under the first estimate and real under the second.

Worked example

Save, load and evaluate

Save the weights, load them into a fresh model and check the two give the same logits. The page's torch.save writes to its own in-memory files.

Uses the setup above, 57 lines.

model, train, val, biggest = run(3e-2, steps=40)
torch.save(model.state_dict(), "ckpt.pt")
fresh = GPT(len(vocab), 24, 32, 2, 4)
fresh.load_state_dict(torch.load("ckpt.pt"))
model.eval()
fresh.eval()
x, y = get_batch(val_data, batch=2)
print("same logits:", torch.allclose(model(x), fresh(x)))
print("tensors saved:", len(model.state_dict()))
In the lesson you can change this and run it.

The state_dict holds every parameter by name, 30 tensors for this model, and the fresh model gives the same logits once it has loaded them. eval() switches off dropout and batch-norm updates in models that have them; this small GPT has neither, so it changes nothing here, but it matters for most models, so training code calls eval() for every validation pass and train() after.

Reading the curves: a learning-rate sweep

The same run at three peak rates, ten times apart. The setup's run returns the largest gradient norm it clipped, and the final validation loss, so the cell can print both and draw the three training curves.

Worked example

Three learning rates, one schedule

Peak rates 0.003, 0.03 and 0.3, each for 100 steps with warmup and a clip of 1.0.

Uses the setup above, 57 lines.

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6.5, 3))
for peak in (3e-3, 3e-2, 3e-1):
    model, train, val, biggest = run(peak)
    print(f"peak {peak:<6} final validation loss {val[-1][1]:.3f}  "
          f"largest gradient norm {biggest:.2f}")
    ax.plot(train, label=f"peak {peak}")
ax.set_xlabel("step")
ax.set_ylabel("training loss")
ax.legend()
In the lesson you can change this and run it.

The middle rate wins, with a final validation loss of 1.015. At 0.003 the model learns steadily but too slowly for 100 steps and ends at 2.454. At 0.3 the loss stays near 3.5 and the gradient norm reaches 3.54, well above the limit, so clipping is firing on many steps and the model is not learning: a rate that is too high looks like a flat curve and a large gradient norm. The best rate depends on the model; a bigger GPT wants a smaller one.

Quick check
The validation estimate uses 10 batches of 32 windows, 24 tokens each. A guess: how many next-token guesses does it average over?
print(10 * 32 * 24)
  1. 240
  2. 320
  3. 768
  4. 7680
Show the answer

7680

Each batch grades 32 x 24 = 768 guesses, and ten batches grade 7,680. A larger count gives a steadier average, which is the point of the estimate.

Quick check
A run's training loss is flat near 3.5 after 100 steps, and the logged gradient norm is often above the clip limit. What do you try first?
  1. Lower the peak learning rate
  2. Train for more steps at the same rate
  3. Switch the validation estimate to one batch
  4. Raise the clip limit so nothing is clipped
Show the answer

Lower the peak learning rate

A flat loss with large, often-clipped gradients means the steps are too big. Lowering the peak rate is the first change. Raising the clip limit would let the oversized steps through.

In your own words
In the sweep, what told you that the highest rate was too high, before you looked at the final loss?

Write a sentence or two, then compare it with a good answer in the lesson.

In real models: the same loop at scale

A public GPT training script has the shape you just wrote, with a config at the top: learning_rate, warmup_iters, lr_decay_iters, min_lr, grad_clip, eval_interval and eval_iters. Each is one of the numbers in this lesson. Two more pieces appear only at scale, and the card below shows both: mixed precision, which runs the matrix products in 16 bits to save memory and time, and torch.compile, which turns the model into faster fused code.

Where this shows up in an engineer's week

On the job: log, at every step or every few, the training loss, the learning rate, the gradient norm and the tokens per second, and at every evaluation the validation loss over a fixed set of batches. Save a checkpoint with the optimizer's state and the step, so a crash resumes instead of restarting. When a run goes wrong, read these four numbers together: a loss that jumps while the gradient norm spikes points at a bad batch or a rate that is too high, and a loss that is flat with a tiny norm points at a rate that is too low.

Goes deeper in

In your own words
A teammate's run diverges at step 3,000 with a loss spike. Which logged numbers do you look at, and what might you change?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 9 review cards for spaced repetition

Try a quiz question

A training loss has a single huge spike, then recovers. Which habit limits the damage a spike like that does to the weights?

  1. Gradient clipping
  2. A longer warmup
  3. More validation batches
  4. A larger batch of text

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 5 · about 36 hours of study

Transformers

Attention, GPT from scratch and pretraining 47 lessons 1 free
  1. Language modeling FreePredict the next token, and score it with cross-entropy.30 min
  2. Tokenization and BPEHow tokenizers are built from data.40 min
  3. A bigram modelThe simplest language model, built from counts.35 min
  4. Sampling strategiesTemperature, top-k and top-p: turning probabilities into text.35 min
  5. Beam search and constrained decodingWhy the likeliest word isn't the likeliest sentence, beam search with log probabilities, its taste for short and repetitive text, and forcing output into a grammar: what that guarantees and what it doesn't.40 min
  6. Neural language modelsWhy counting fails, and what embeddings fix.35 min
  7. Attention: the intuitionEach token decides which other tokens matter to it.30 min
  8. Queries, keys and valuesThe three projections behind attention.35 min
  9. Scaled dot-product attentionThe attention formula, line by line.35 min
  10. Causal maskingStop tokens from seeing the future.25 min
  11. Multi-head attentionSeveral attention patterns in parallel.40 min
  12. Positional encodingsTell the model where each token sits.30 min
  13. The transformer blockAttention, MLP, residuals and norms together.40 min
  14. The GPT architectureA stack of decoder blocks that predicts the next token.40 min
  15. Build a tiny GPTWrite and train a small GPT from scratch.45 min
  16. From your GPT to GPT-2 in Hugging FaceMatch your GPT's tensors to GPT-2's names and shapes, count its 124,439,808 parameters, load renamed weights into your own class, pad a batch and read a config.40 min
  17. Training a GPT wellWarmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.35 min
  18. Encoders and embedding modelsBERT-style models and where they fit.40 min
  19. Tokenization: text to integer IDsFollow "The cat sat" into the three integers a model receives.30 min
  20. Embeddings and positional wavesFrom three IDs to the 3 × 8 matrix that enters the first block.35 min
  21. Inside the attention matrixEvery entry of the 3 × 3 attention matrix, computed and steered.40 min
  22. The feed-forward layerExpand to four times d_model and back: shapes, parameters and what the layer stores.35 min
  23. The LM head and samplingFrom the last row to a probability for every token, and the next word.35 min
  24. Pretraining at scaleData, compute and what models learn from the internet.40 min
  25. Scaling lawsHow loss falls predictably with size, data and compute.40 min
  26. Inference and the KV cacheWhy generation is fast after the first token.40 min
  27. The block in today's open models OptionalRMSNorm, SwiGLU, rotary positions and grouped-query attention swapped into the block you built, with the parameter and cache counts before and after.35 min
  28. Long contextEfficient attention and million-token windows.35 min
  29. Mixture of expertsRoute each token to a few specialist sub-networks.35 min
  30. Capstone: diagnose your own GPTEntropy per position, a head's attention pattern and a head ablation on the tiny GPT, then the same three measurements on a GPT you train yourself.45 min
  31. Jacobians: derivatives with shapes OptionalThe derivative of a vector function is a matrix; the chain rule multiplies them, and backprop multiplies by them without building them.35 min
  32. Backprop through a linear layer OptionaldX = dY Wᵀ, dW = Xᵀ dY, db = column sums: derived by shape and by entry, checked on the tiny GPT's own projection.35 min
  33. The softmax Jacobian and p − y OptionalEvery output depends on every input: the full Jacobian, its cheap vector product, and why cross-entropy's gradient is p − y.35 min
  34. Backprop through attention OptionalFive formulas from the output back to queries, keys and values; why masked scores get zero gradient; what FlashAttention recomputes.40 min
  35. Multimodal transformersHow models see images and hear audio.35 min
  36. Object detection: boxes, overlap and suppressionEvery object and where it is: boxes in two formats, intersection over union, why one object gets many boxes, non-maximum suppression and its threshold, and grading with average precision.45 min
  37. Segmentation: a label for every pixelMasks, and how to grade them with IoU and Dice; why a pixel's label needs context and too much context blurs; the shrink-then-enlarge network with skip connections; and turning a mask into countable objects.45 min
  38. Build a vision transformerPatch embeddings, a class token, position embeddings and an unmasked block, trained in the page; then two failures that explain it: no positions means no layout, and few images means losing to a CNN a fraction of its size.45 min
  39. Image-text models in depth: retrieval, negatives and limitsHow to test a shared image-text space with recall at k, what the rest of the batch teaches, what hard and false negatives do, and why a similarity score can't see word order, counts or "no".45 min
  40. Self-supervised learning: labels from the data itselfPretrain an encoder with no labels by matching two views of each image, test it with a linear probe, and see what the augmentations remove, what collapse looks like, and how filling in masked parts differs.45 min
  41. Diffusion 1: turning data into noiseThe easy half of a diffusion model: one step of noise, why the data is shrunk as noise is added, the jump to any step with alpha bar, how to read a noise schedule, and the training examples a denoiser learns from.40 min
  42. Diffusion 2: learning to denoise, and samplingTrain a network to predict noise, then run it a hundred times from pure noise and watch a ring assemble: the sampling loop, why fresh noise goes back in, fewer steps and where they break, and a check that samples are new.45 min
  43. Diffusion 3: images, labels and guidanceFrom points to small images, a label the denoiser can be asked with, and classifier-free guidance: why a model that is told what to draw still often doesn't, how the guidance scale fixes it, and what it breaks when turned up too far.45 min
  44. Diffusion 4: latent diffusion and the whole pipelineWhy image generators diffuse an autoencoder's codes and not pixels: a latent diffusion model in miniature, what the autoencoder must be trained to do, the three parts of a text-to-image system, and where the cost of an image goes.40 min
  45. Sound as numbers: waveforms and spectrogramsA recording as an array of samples, why the sampling rate must be twice the highest pitch, spectrograms and the window trade-off, and the log-mel features a speech model is given.40 min
  46. Recognition and synthesis as a pipelineA whole speech recognizer for a made-up language: features, a guess per frame, collapsing repeats and blanks, a dictionary to repair mistakes, word error rate, and synthesis as the same stages run backward.40 min
  47. The voice agent loop and its latency budgetOne turn of a voice agent, timed: why every stage must stream, how to decide that someone has finished speaking, what to do when they interrupt, and why the slow turns matter more than the average.45 min
Part of a free course 22 lessons

Build an LLM from scratch: a free course

Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write