A training script, start to finish
Config, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.
What it covers6
Why it matters
A model trained in a notebook is a demonstration. A model trained by a script is a product: someone else can run it, and so can you in six months, on a bigger machine, after a crash at step 4,000. The difference is not new mathematics. It is discipline: every choice written down in one place, every random number seeded, and a checkpoint that holds everything the run needs to carry on as if it had never stopped. Teams lose days to runs that "resumed" with a fresh optimizer and quietly became a different experiment.
What you'll be able to do
- Name the seven parts of a training script and the one thing each hands to the next
- List what a checkpoint must hold to resume a run exactly: model, optimizer, schedule position, step and random state
- Stop a run at step 50, resume it from bytes, and match an unbroken run, then break the match three ways
- Write a module with buffers and a ModuleList, a Dataset with a collate function, and one device variable
- Save and load a checkpoint with torch.save and torch.load, and say what weights_only and map_location do
The idea: a script you can stop and trust
This lesson rests on Training at scale: precision, accumulation, data and many GPUs, and uses the pieces this phase has built: a Dataset, a module, a cosine schedule and a best.pt file. Capstone: train a seam-tear spotter, next, puts them to work. First, what the optimizer remembers.
Show the answer
A running average of the slope, and a running average of its square
Adam keeps m, the running average of the gradient, and v, the running average of the squared gradient. Each weight's step is m divided by the square root of v. These are the optimizer's memory, and they live nowhere in the model.
A training script has seven parts. Each one hands a thing to the next, and each one has a way to go wrong that the one before can't see.
- Config. Every choice in one place: learning rate, batch size, steps, seed. A run you can't re-run is an anecdote.
- Seed. Every source of randomness gets a seed from the config: the shuffle, the initial weights, dropout.
- Data. A
Datasetthat returns one example and aDataLoaderthat stacks them into batches, with the split made before any training. - Model. An
nn.Module, built from the config, moved to onedevice. - Optimizer and schedule. The update rule and the learning rate's path over the steps.
- Loop. Batch, loss, backward, step. Count the steps.
- Checkpoint. A file that lets the run carry on from where it stopped.
Step through the seven parts. At each one, the picture says what it hands on.
Seven parts, in the order a script runs them.
What a checkpoint must hold
Everything that changes while the run goes on, and nothing that can be rebuilt from the config:
- Model state. The weights, and any buffers.
- Optimizer state. Adam's
mandvfor every weight. That is two more numbers per weight, so Adam's state is twice the size of the model. - Schedule position. Where the learning rate is on its path. Usually that is just the step.
- Step. The one counter the loop owns.
- Random state. Where the stream that picks batches had got to.
Leave one out and the run still goes on. That is what makes it dangerous: nothing crashes. The warmup below is the schedule as the lesson's script uses it. It reads the step, and nothing else.
Show the answer
The order of the batches, so the two runs take different paths
Every source of randomness needs its seed. The starting weights match, but the shuffle gives each run a different batch order, so the paths and the final weights differ. A seed that covers only part of the randomness is worse than none, since it looks like control.
import math peak, warmup, total = 0.1, 10, 100 def lr_at(step): if step < warmup: return peak * (step + 1) / warmup t = (step - warmup) / (total - warmup) return peak * 0.5 * (1 + math.cos(math.pi * t)) print(round(lr_at(0), 3), round(lr_at(9), 3), round(lr_at(10), 3))
Show the answer
0.01 0.1 0.1
Warmup counts step + 1 of 10: step 0 gives 0.1 × 1 / 10 = 0.01, and step 9 reaches 0.1 × 10 / 10 = 0.1. At step 10 the cosine starts, at t = 0, where 0.5 × (1 + 1) = 1, so the rate is 0.1 again. The rate is a function of the step. That is why a lost step number restarts the warmup.
Write a sentence or two, then compare it with a good answer in the lesson.
From scratch: stop at step 50, resume, match
A claim you can test: a run stopped at step 50 and resumed from a file should end exactly where an unbroken run of 100 steps ends. First, the idea on one weight. The cell holds a weight w, a momentum v and a step count, saves all three to bytes, reads them back as a new process would, and carries on.
Now the real run. Run in this lesson's setup block is a whole script in miniature: a config, data made from the seed, Adam written out, and a warmup then cosine schedule. It fits a line to 200 noisy points. The setup is folded above the first cell. First, the unbroken run.
Setup (runs before every cell) · 65 lines
The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.
import io import math import pickle import numpy as np import torch import torch.nn as nn from torch.utils.data import Dataset, DataLoader CONFIG = {"seed": 0, "rows": 200, "batch": 20, "lr": 0.1, "warmup": 10, "steps": 100} def make_data(seed, rows): """Three inputs and a constant 1 for the bias. True weights: 1.5, -2, 0.5, 0.3.""" rng = np.random.default_rng(seed) x = rng.normal(size=(rows, 3)) y = x @ np.array([1.5, -2.0, 0.5]) + 0.3 + rng.normal(scale=0.1, size=rows) return np.hstack([x, np.ones((rows, 1))]), y def lr_at(step, cfg): """Warm up for cfg["warmup"] steps, then fall along half a cosine to zero.""" if step < cfg["warmup"]: return cfg["lr"] * (step + 1) / cfg["warmup"] t = (step - cfg["warmup"]) / (cfg["steps"] - cfg["warmup"]) return cfg["lr"] * 0.5 * (1 + math.cos(math.pi * t)) class Run: """Linear regression trained with Adam, written out, with every part of a script.""" def __init__(self, cfg): self.cfg = cfg self.x, self.y = make_data(cfg["seed"], cfg["rows"]) self.rng = np.random.default_rng(cfg["seed"] + 1) # picks each batch self.w, self.m, self.v = np.zeros(4), np.zeros(4), np.zeros(4) self.step = 0 def loss(self): return float(((self.x @ self.w - self.y) ** 2).mean()) def train(self, until): c = self.cfg while self.step < until: rows = self.rng.choice(c["rows"], size=c["batch"], replace=False) x, y = self.x[rows], self.y[rows] g = 2 * x.T @ (x @ self.w - y) / len(y) self.m = 0.9 * self.m + 0.1 * g # Adam's two memories self.v = 0.999 * self.v + 0.001 * g ** 2 t = self.step + 1 m_hat, v_hat = self.m / (1 - 0.9 ** t), self.v / (1 - 0.999 ** t) self.w = self.w - lr_at(self.step, c) * m_hat / (np.sqrt(v_hat) + 1e-8) self.step += 1 return self def state(self): return {"step": self.step, "w": self.w, "m": self.m, "v": self.v, "rng": self.rng.bit_generator.state} def load(self, state, skip=()): """Put a saved state back. Names in skip stay as a new run has them.""" if "step" not in skip: self.step = state["step"] if "w" not in skip: self.w = state["w"].copy() if "m" not in skip: self.m, self.v = state["m"].copy(), state["v"].copy() if "rng" not in skip: self.rng.bit_generator.state = state["rng"] return self
whole = Run(CONFIG) print("loss at the start:", round(whole.loss(), 4)) whole.train(CONFIG["steps"]) print("loss after", whole.step, "steps:", round(whole.loss(), 4)) print("weights:", np.round(whole.w, 3), "(the true ones: 1.5, -2, 0.5, 0.3)")
Now stop at step 50 and save a checkpoint. pickle.dumps turns the state into bytes, the way torch.save does for tensors. Then build a new Run from the config, as a new process would, load the bytes, and finish. The cell runs four resumes: with everything, then each time with one part left out.
Uses the setup above, 65 lines.
whole = Run(CONFIG).train(100) first = Run(CONFIG).train(50) blob = pickle.dumps(first.state()) print("checkpoint is", type(blob).__name__, "holding", sorted(pickle.loads(blob))) def resume(skip=()): run = Run(CONFIG) # a new process, built from the config run.load(pickle.loads(blob), skip) return run.train(run.step + 50) # 50 steps are still to do cases = [("everything", ()), ("no optimizer state", ("m",)), ("no random state", ("rng",)), ("no step", ("step",))] for name, skip in cases: run = resume(skip) gap = float(np.abs(run.w - whole.w).max()) print(name.ljust(20), "steps", run.step, "largest weight gap", f"{gap:.6f}")
With everything saved the gap is exactly 0.000000: the resumed run is the unbroken run, to every digit and not only to four places. Leave out any one part and the weights end up a little different, by 0.076 without Adam's memory, by 0.011 without the random state and by 0.070 without the step. None of them crashes. The final loss of each is close to the others, since this is a small, easy problem, so a loss curve would not show the mistake. The gap in the weights does, and only because we kept the unbroken run to compare with. A resumed run that isn't the same run is a different experiment.
The step deserves a picture. The schedule reads the step, so a lost step sends the learning rate back to the beginning of its warmup, in the middle of the run.
Uses the setup above, 65 lines.
import matplotlib.pyplot as plt steps = range(100) kept = [lr_at(s, CONFIG) for s in steps] lost = kept[:50] + [lr_at(s - 50, CONFIG) for s in range(50, 100)] plt.plot(steps, kept, label="step saved") plt.plot(steps, lost, linestyle="--", label="step lost at 50") plt.axvline(50, color="gray", linestyle=":") plt.xlabel("step") plt.ylabel("learning rate") plt.legend() print("rate at step 50:", round(lr_at(50, CONFIG), 4), "kept, against", round(lr_at(0, CONFIG), 4), "when the step is lost")
Show the answer
The weights, Adam's m and v, the step and the random state
All four of the cell's variants that left something out ended with a gap above 0. The config is rebuilt from its own file, but it holds the starting values. The weights, m and v, the step and the random stream all changed during the run, so all of them were saved.
import pickle state = {"step": 5, "w": [0.5, 1.5]} blob = pickle.dumps(state) state["w"][0] = 99 print(pickle.loads(blob)["w"][0], type(blob).__name__)
Show the answer
0.5 bytes
pickle.dumps copies the state into bytes at that moment. Changing the dictionary afterwards doesn't change the bytes, so loading them gives 0.5. That is what a checkpoint is for: a snapshot that training can't disturb.
Worked example: what a checkpoint holds, in bytes
The run saved at step 50 holds its weights, Adam's two memories and the random state. Each of the four weights is a float64, 8 bytes. The cell counts what each part costs.
Uses the setup above, 65 lines.
run = Run(CONFIG).train(50) state = run.state() for key in ("w", "m", "v"): print(key, state[key].nbytes, "bytes") print("model and optimizer:", sum(state[k].nbytes for k in "wmv"), "bytes") print("the model alone:", state["w"].nbytes, "bytes") print("the whole file:", len(pickle.dumps(state)), "bytes, with step and stream")
The four weights take 32 bytes, and so do m and v: three sets, 96 bytes in all, three times the model alone. The file is larger than 96, since it also holds the step, the random stream's state and the pickle's own bookkeeping. At this size none of it matters. For a model with a billion weights, Adam's two memories are the larger share of the file.
The learning rate at a step
Write lr_at(step, peak, warmup, total): for step < warmup the rate is peak * (step + 1) / warmup. From warmup on it falls along half a cosine, peak * 0.5 * (1 + cos(pi * t)), where t = (step - warmup) / (total - warmup). Use math.
Uses the setup above, 65 lines.
import math def lr_at(step, peak, warmup, total): return 0.0 print(round(lr_at(0, 0.1, 10, 100), 4), round(lr_at(55, 0.1, 10, 100), 4))
How big is a checkpoint?
Write checkpoint_bytes(n_params, optimizer, bytes_per=4). The model is n_params * bytes_per bytes. The optimizer adds more sets of the same size: "sgd" adds none, "momentum" adds 1 (the velocity), "adam" adds 2 (m and v). Raise a ValueError for any other name.
Uses the setup above, 65 lines.
def checkpoint_bytes(n_params, optimizer, bytes_per=4): return 0 print(checkpoint_bytes(1000, "adam"), checkpoint_bytes(1000, "sgd"))
Write a sentence or two, then compare it with a good answer in the lesson.
In PyTorch: module, dataset, checkpoint
Real scripts do all of this with objects PyTorch provides. A module, a dataset and a device run on this page. The optimizer's saved state needs real PyTorch, so it comes as a card.
A module with buffers, one device and a round trip
A buffer is a tensor the model owns that training doesn't change: a mean, a mask, a position table. register_buffer tells the module to move it with the model and save it in the checkpoint. A plain attribute is neither. nn.ModuleList holds a list of layers so that PyTorch can see them. A plain Python list hides them from parameters(), and the optimizer then never updates them.
A script also defines the device once and uses that one name everywhere: for the model, for each batch and in torch.load's map_location. This page has no GPU, so the line below picks the CPU. On a machine with one it picks the GPU. The cell saves a model and a step into an in-memory file, loads them into a new model built with the wrong mean and spread, and checks that the buffers came back with the weights.
Uses the setup above, 65 lines.
class Net(nn.Module): def __init__(self, mean, std, width=8, depth=2): super().__init__() self.register_buffer("mean", mean) # saved and moved, never trained self.register_buffer("std", std) sizes = [len(mean)] + [width] * depth self.layers = nn.ModuleList( [nn.Linear(sizes[i], sizes[i + 1]) for i in range(depth)]) self.head = nn.Linear(width, 1) def forward(self, x): x = (x - self.mean) / self.std for layer in self.layers: x = torch.relu(layer(x)) return self.head(x).squeeze(1) class Hidden(nn.Module): def __init__(self): super().__init__() self.layers = [nn.Linear(3, 3), nn.Linear(3, 3)] # a plain list device = "cuda" if torch.cuda.is_available() else "cpu" torch.manual_seed(1) model = Net(torch.tensor([1.0, 2.0, 3.0]), torch.tensor([2.0, 2.0, 2.0])).to(device) print("parameters:", [name for name, p in model.named_parameters()]) print("buffers: ", [name for name, b in model.named_buffers()]) print("saved: ", list(model.state_dict())) print("parameters in a plain list of layers:", len(list(Hidden().parameters()))) x = torch.randn(5, 3).to(device) file = io.BytesIO() torch.save({"step": 50, "model": model.state_dict()}, file) file.seek(0) ckpt = torch.load(file, map_location=device, weights_only=True) fresh = Net(torch.zeros(3), torch.ones(3)).to(device) # wrong buffers on purpose print("before loading, same output:", torch.allclose(model(x), fresh(x))) fresh.load_state_dict(ckpt["model"]) print("after loading, same output:", torch.allclose(model(x), fresh(x))) print("mean came back:", fresh.mean.tolist(), "| step:", ckpt["step"])
The buffers are saved but are not parameters, so the optimizer never sees them. The plain list hides all its weights: 0 parameters, so nothing would train. The fresh model gave other answers until it was loaded. Had the mean been a plain attribute, the file would not have held it, and a resumed model would have standardized its inputs with the wrong numbers, silently.
A Dataset and a collate function
A Dataset returns one example. A collate function turns a list of them into one batch. The default stacks tensors, which fails for different lengths. This one pads the short ones and returns a mask, so the loss can ignore the padding.
Uses the setup above, 65 lines.
class Tokens(Dataset): def __init__(self, seqs): self.seqs = seqs def __len__(self): return len(self.seqs) def __getitem__(self, i): return torch.tensor(self.seqs[i]), len(self.seqs[i]) % 2 def collate(batch): longest = max(len(ids) for ids, label in batch) padded = torch.zeros(len(batch), longest, dtype=torch.long) mask = torch.zeros(len(batch), longest, dtype=torch.bool) for row, (ids, label) in enumerate(batch): padded[row, :len(ids)] = ids mask[row, :len(ids)] = True return padded, mask, torch.tensor([label for ids, label in batch]) data = Tokens([[5, 6, 7], [8, 9], [3, 4, 5, 6, 7], [1]]) padded, mask, labels = next(iter(DataLoader(data, batch_size=4, collate_fn=collate))) print("padded", tuple(padded.shape), "mask", tuple(mask.shape)) print("labels", labels.tolist()) print(padded.tolist()) print("real tokens per row:", mask.sum(dim=1).tolist())
batch = [[1, 2, 3], [4], [5, 6]] longest = max(len(ids) for ids in batch) real = sum(len(ids) for ids in batch) print(longest, real, longest * len(batch))
Show the answer
3 6 9
The longest has 3 tokens. There are 3 + 1 + 2 = 6 real tokens. Padding every row to 3 makes 3 × 3 = 9 slots, so 3 are padding, and the mask marks them. Sorting sequences by length before batching cuts that waste.
import torch import torch.nn as nn class Scaler(nn.Module): def __init__(self): super().__init__() self.register_buffer("mean", torch.zeros(3)) self.layer = nn.Linear(3, 2) m = Scaler() print(len(list(m.parameters())), len(list(m.buffers())))
Show the answer
2 1
The linear layer has two parameters, its weight and its bias. The mean is one buffer. The optimizer is handed the two parameters, and the checkpoint holds all three tensors.
One more format is worth knowing by name. safetensors is a library with its own file format, which holds tensors and nothing else, so loading a file can never run code. Model hubs use it for shared weights. It stores the model's tensors only, so a training checkpoint still goes through torch.save, with the optimizer and the step beside the weights.
On the job
What goes wrong with scripts in a real week:
- Resume without checking. Run this lesson's test: stop a short run in the middle, resume it, and compare the end with an unbroken run. It takes a minute, and it catches the missing optimizer.
- A half-written checkpoint. A crash mid-save leaves a broken file. Write to a temporary name, then rename it over the old one. A rename is all or nothing.
- Keep more than one. Keep the last few and the best. A run can go wrong at step 3,000 and show it at 3,200.
- Config in the file. Save the config beside the weights, so the file says how it was made.
- A different batch on resume. Fewer GPUs means a different batch, learning rate and steps per epoch. Check the config you resume under against the saved one.
- Where this shows up. Debugging training reads a failing run from its numbers. A resume that quietly changed the run is the hardest to read, since every number looks plausible.
Goes deeper in: Capstone: train a seam-tear spotter, which builds the Dataset, the module, the cosine schedule and best.pt for a real task, Training at scale: precision, accumulation, data and many GPUs for accumulation and precision, and Optimizers for the memory Adam keeps.
Write a sentence or two, then compare it with a good answer in the lesson.
Practice what you just read
- 6 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 14 review cards for spaced repetition
Try a quiz question
A run is resumed from a file that holds only the model's weights. What happens?
Before this lesson
Where this lesson leads.
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
Phase 4 · about 24 hours of study
Deep learning
Neural networks, backprop and training them well
31 lessons 1 free
- The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
- Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
- Multi-layer networksStack layers to learn complex functions.35 min
- The forward passFollow numbers through a network to a prediction.36 min
- Computation graphsEvery calculation as a graph of small operations.40 min
- BackpropagationThe chain rule applied backwards to get every gradient.42 min
- Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
- PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
- PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
- The training loopForward, loss, backward, step, repeat.40 min
- Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
- OptimizersMomentum and Adam.41 min
- Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
- Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
- Weight initializationStart weights at the right scale.33 min
- Normalization layersBatch and layer norm keep training stable.40 min
- Residual connectionsSkip connections that let gradients flow.40 min
- DropoutRandomly switch off neurons to prevent overfitting.35 min
- Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
- Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
- Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
- Convolutional networksPattern detectors for images.44 min
- Embedding layersLearned lookup tables that turn tokens into vectors.38 min
- Recurrent networksProcess sequences step by step, and their limits.43 min
- LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
- Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
- AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
- GPUs and computeWhy deep learning runs on graphics cards.40 min
- Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
- A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
- Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write