Build an LLM from scratch: a free course
Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.
Warmup and cosine decay, gradient clipping, a validation loss you can trust, saving and loading, and two training mistakes read off the loss curve.
The tiny GPT you built trained in 300 steps with a fixed learning rate. A real run is thousands of steps, and four habits keep it healthy: a warmup, a decaying rate, a clip on the gradient and a validation loss that you can trust. They are small changes to the loop you already wrote, and each has a number you can read in the log.
This lesson recalls the schedule and the training script from Phase 4 and teaches only what is particular to a GPT. You will write the schedule and the clip, train the model with them, watch a sweep of learning rates, and read the curves.
print(3e-3 * (4 + 1) / 10)
0.0015
The warmup rule is peak x (step + 1) / warmup, so step 4 gives 0.003 x 5 / 10 = 0.0015: halfway up, because steps are counted from 0.
This rests on Learning-rate schedules in PyTorch, which built warmup and cosine decay by hand, and on A training script, start to finish, where a training run had a config, a seed and a checkpoint. Here is only what is particular to a GPT, on the model you built in Build a tiny GPT.
A 300-step demo with a fixed learning rate trains a GPT, but a real run is long, and four things keep it healthy.
The setup's lr_at is warmup then cosine to a floor. Ten steps of warmup, 100 in all, peak 0.003.
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import math import re import requests import torch import torch.nn as nn from tinygpt import GPT text = requests.get("https://api.gradient.test/v1/corpus").json()["text"] tokens = re.findall(r"[A-Za-z]+|[0-9]+|[^\sA-Za-z0-9]|\n", text) vocab = sorted(set(tokens)) stoi = {t: i for i, t in enumerate(vocab)} data = torch.tensor([stoi[t] for t in tokens]) split = int(0.9 * len(data)) train_data, val_data = data[:split], data[split:] def get_batch(d, block=24, batch=16): starts = torch.randint(0, len(d) - block - 1, (batch,)).tolist() x = torch.stack([d[i:i + block] for i in starts]) y = torch.stack([d[i + 1:i + block + 1] for i in starts]) return x, y def loss_of(model, x, y): return nn.functional.cross_entropy(model(x).view(-1, len(vocab)), y.view(-1)) def lr_at(step, peak, warmup, total): if step < warmup: return peak * (step + 1) / warmup progress = (step - warmup) / (total - warmup) return 0.1 * peak + 0.9 * peak * 0.5 * (1 + math.cos(math.pi * progress)) @torch.no_grad() def estimate_loss(model, d, iters=10, batch=32): model.eval() total = sum(loss_of(model, *get_batch(d, batch=batch)).item() for _ in range(iters)) model.train() return total / iters def run(peak, steps=100, warmup=10, clip=1.0, every=20, seed=0): torch.manual_seed(seed) model = GPT(len(vocab), 24, 32, 2, 4) opt = torch.optim.AdamW(model.parameters(), lr=peak, weight_decay=0.01) sched = torch.optim.lr_scheduler.LambdaLR( opt, lambda s: lr_at(s, peak, warmup, steps) / peak) train, val, biggest = [], [], 0.0 for step in range(steps): x, y = get_batch(train_data) loss = loss_of(model, x, y) opt.zero_grad() loss.backward() norm = nn.utils.clip_grad_norm_(model.parameters(), clip) biggest = max(biggest, float(norm)) opt.step() sched.step() train.append(loss.item()) if step % every == 0 or step == steps - 1: val.append((step, estimate_loss(model, val_data))) return model, train, val, biggest
print([round(lr_at(s, 3e-3, 10, 100), 6) for s in range(0, 100, 10)]) print(round(lr_at(99, 3e-3, 10, 100), 6))
The rate starts at 0.0003, a tenth of the peak, reaches 0.003 at step 10, and then falls along the cosine: 0.001884 at step 50, 0.000381 at step 90 and 0.000301 at the last step, a little above the floor of 0.0003.
The values printed above, every 10 steps: a short climb, then the cosine down toward a tenth of the peak.
It keeps the first noisy updates small while estimates settle
In the first steps the optimizer's gradient statistics are unreliable, so a small learning rate keeps early updates safe. The rate then climbs to its peak.
0.2
Clipping multiplies the gradient by limit / length = 1 / 5 = 0.2, so the new length is exactly 1.0 and the direction is unchanged.
Write a sentence or two, then compare it with a good answer in the lesson.
The lesson's setup, folded at the first cell, loads the practice corpus, cuts it into windows, and defines lr_at, estimate_loss and run, the training loop. The three lines that matter are these:
fn = lambda s: lr_at(s, peak, warmup, steps) / peak sched = torch.optim.lr_scheduler.LambdaLR(opt, fn) norm = nn.utils.clip_grad_norm_(model.parameters(), clip) # between backward and step sched.step() # after opt.step()
LambdaLR multiplies the optimizer's base rate by a function of the step, which is why the function returns a fraction of the peak. clip_grad_norm_ rescales every gradient in place and returns the length it measured before. The loop also estimates the validation loss every 20 steps.
A 2-layer GPT, 100 steps, a peak rate of 0.03, which this small model likes. Training loss every step, validation loss estimated over 10 batches every 20 steps.
Uses the setup above, 57 lines.
model, train, val, biggest = run(3e-2) for step, v in val: print(f"step {step:>3} validation loss {v:.3f}") print("final training loss", round(train[-1], 3)) print("largest gradient norm", round(biggest, 2)) import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(6.5, 3)) ax.plot(train, label="training (every step)") steps = [s for s, v in val] ax.plot(steps, [v for s, v in val], "o-", label="validation (10 batches)") ax.set_xlabel("step") ax.set_ylabel("loss") ax.legend()
The validation loss falls from 5.336 at step 0 to about 1.0 by the end, and the training loss ends near 0.95. The two curves track each other, which says the model is not memorizing yet. The largest gradient norm was 0.73, under the limit of 1.0, so clipping never fired on this run: it is insurance, and the next experiment shows it working.
LambdaLR multiplies the optimizer's base rate by a function of the step. With base rate 0.03 and the schedule above, what factor does the function return at step 0? A guess: what does this print?import math def lr_at(step, peak, warmup, total): if step < warmup: return peak * (step + 1) / warmup return peak print(round(lr_at(0, 3e-2, 10, 100) / 3e-2, 2))
0.1
At step 0 the schedule gives 0.03 x 1 / 10 = 0.003, which is 0.1 of the base rate, so the factor is 0.1.
Write lr_schedule(step, peak, warmup, total): during the first warmup steps the rate is peak * (step + 1) / warmup. After that it falls along half a cosine from peak at step warmup to 0.1 * peak at step total: 0.1 * peak + 0.9 * peak * 0.5 * (1 + cos(pi * progress)), where progress is (step - warmup) / (total - warmup).
Uses the setup above, 57 lines.
import math def lr_schedule(step, peak, warmup, total): return peak print(round(lr_schedule(4, 3e-3, 10, 100), 6))
Write clip_scale(norm, max_norm): the factor to multiply a gradient by so that its length is at most max_norm. If the gradient is already short enough, the factor is 1.0; otherwise it is max_norm / norm.
Uses the setup above, 57 lines.
def clip_scale(norm, max_norm): return 1.0 print(clip_scale(5.0, 1.0))
The same kind of trained model, scored eight times with one batch and eight times with ten. The model is fixed; only the estimate changes.
Uses the setup above, 57 lines.
model, train, val, biggest = run(3e-2) torch.manual_seed(5) one = [estimate_loss(model, val_data, iters=1) for _ in range(8)] ten = [estimate_loss(model, val_data, iters=10) for _ in range(8)] print("one batch: ", round(min(one), 3), "to", round(max(one), 3)) print("ten batches:", round(min(ten), 3), "to", round(max(ten), 3))
One batch of 32 windows swings between 0.955 and 1.081 for the same model, a gap of 0.126. Ten batches narrow it to 0.990 to 1.023, a gap of 0.033. A change of 0.05 between two runs would be noise under the first estimate and real under the second.
Save the weights, load them into a fresh model and check the two give the same logits. The page's torch.save writes to its own in-memory files.
Uses the setup above, 57 lines.
model, train, val, biggest = run(3e-2, steps=40) torch.save(model.state_dict(), "ckpt.pt") fresh = GPT(len(vocab), 24, 32, 2, 4) fresh.load_state_dict(torch.load("ckpt.pt")) model.eval() fresh.eval() x, y = get_batch(val_data, batch=2) print("same logits:", torch.allclose(model(x), fresh(x))) print("tensors saved:", len(model.state_dict()))
The state_dict holds every parameter by name, 30 tensors for this model, and the fresh model gives the same logits once it has loaded them. eval() switches off dropout and batch-norm updates in models that have them; this small GPT has neither, so it changes nothing here, but it matters for most models, so training code calls eval() for every validation pass and train() after.
The same run at three peak rates, ten times apart. The setup's run returns the largest gradient norm it clipped, and the final validation loss, so the cell can print both and draw the three training curves.
Peak rates 0.003, 0.03 and 0.3, each for 100 steps with warmup and a clip of 1.0.
Uses the setup above, 57 lines.
import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(6.5, 3)) for peak in (3e-3, 3e-2, 3e-1): model, train, val, biggest = run(peak) print(f"peak {peak:<6} final validation loss {val[-1][1]:.3f} " f"largest gradient norm {biggest:.2f}") ax.plot(train, label=f"peak {peak}") ax.set_xlabel("step") ax.set_ylabel("training loss") ax.legend()
The middle rate wins, with a final validation loss of 1.015. At 0.003 the model learns steadily but too slowly for 100 steps and ends at 2.454. At 0.3 the loss stays near 3.5 and the gradient norm reaches 3.54, well above the limit, so clipping is firing on many steps and the model is not learning: a rate that is too high looks like a flat curve and a large gradient norm. The best rate depends on the model; a bigger GPT wants a smaller one.
print(10 * 32 * 24)
7680
Each batch grades 32 x 24 = 768 guesses, and ten batches grade 7,680. A larger count gives a steadier average, which is the point of the estimate.
Lower the peak learning rate
A flat loss with large, often-clipped gradients means the steps are too big. Lowering the peak rate is the first change. Raising the clip limit would let the oversized steps through.
Write a sentence or two, then compare it with a good answer in the lesson.
A public GPT training script has the shape you just wrote, with a config at the top: learning_rate, warmup_iters, lr_decay_iters, min_lr, grad_clip, eval_interval and eval_iters. Each is one of the numbers in this lesson. Two more pieces appear only at scale, and the card below shows both: mixed precision, which runs the matrix products in 16 bits to save memory and time, and torch.compile, which turns the model into faster fused code.
On the job: log, at every step or every few, the training loss, the learning rate, the gradient norm and the tokens per second, and at every evaluation the validation loss over a fixed set of batches. Save a checkpoint with the optimizer's state and the step, so a crash resumes instead of restarting. When a run goes wrong, read these four numbers together: a loss that jumps while the gradient norm spikes points at a bad batch or a rate that is too high, and a loss that is flat with a tiny norm points at a rate that is too low.
Write a sentence or two, then compare it with a good answer in the lesson.
A training loss has a single huge spike, then recovers. Which habit limits the damage a spike like that does to the weights?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Go from counting letter pairs to a small GPT you train yourself: tokenizers, attention, transformer blocks and the GPT architecture, written line by line.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write