Index Ai Academy
Phase 4 · Compute and scale

A training script, start to finish

Config, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.

About 40 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

A model trained in a notebook is a demonstration. A model trained by a script is a product: someone else can run it, and so can you in six months, on a bigger machine, after a crash at step 4,000. The difference is not new mathematics. It is discipline: every choice written down in one place, every random number seeded, and a checkpoint that holds everything the run needs to carry on as if it had never stopped. Teams lose days to runs that "resumed" with a fresh optimizer and quietly became a different experiment.

What you'll be able to do

  • Name the seven parts of a training script and the one thing each hands to the next
  • List what a checkpoint must hold to resume a run exactly: model, optimizer, schedule position, step and random state
  • Stop a run at step 50, resume it from bytes, and match an unbroken run, then break the match three ways
  • Write a module with buffers and a ModuleList, a Dataset with a collate function, and one device variable
  • Save and load a checkpoint with torch.save and torch.load, and say what weights_only and map_location do

The idea: a script you can stop and trust

This lesson rests on Training at scale: precision, accumulation, data and many GPUs, and uses the pieces this phase has built: a Dataset, a module, a cosine schedule and a best.pt file. Capstone: train a seam-tear spotter, next, puts them to work. First, what the optimizer remembers.

Quick check
Adam keeps two numbers for every weight, besides the weight. What are they?
  1. A running average of the slope, and a running average of its square
  2. The weight at the last step, and the weight at the first step
  3. The loss at the last step, and the learning rate
  4. A copy of the batch, and a copy of the gradient
Show the answer

A running average of the slope, and a running average of its square

Adam keeps m, the running average of the gradient, and v, the running average of the squared gradient. Each weight's step is m divided by the square root of v. These are the optimizer's memory, and they live nowhere in the model.

A training script has seven parts. Each one hands a thing to the next, and each one has a way to go wrong that the one before can't see.

  1. Config. Every choice in one place: learning rate, batch size, steps, seed. A run you can't re-run is an anecdote.
  2. Seed. Every source of randomness gets a seed from the config: the shuffle, the initial weights, dropout.
  3. Data. A Dataset that returns one example and a DataLoader that stacks them into batches, with the split made before any training.
  4. Model. An nn.Module, built from the config, moved to one device.
  5. Optimizer and schedule. The update rule and the learning rate's path over the steps.
  6. Loop. Batch, loss, backward, step. Count the steps.
  7. Checkpoint. A file that lets the run carry on from where it stopped.

Step through the seven parts. At each one, the picture says what it hands on.

Try it now

Seven parts, in the order a script runs them.

What a checkpoint must hold

Everything that changes while the run goes on, and nothing that can be rebuilt from the config:

  • Model state. The weights, and any buffers.
  • Optimizer state. Adam's m and v for every weight. That is two more numbers per weight, so Adam's state is twice the size of the model.
  • Schedule position. Where the learning rate is on its path. Usually that is just the step.
  • Step. The one counter the loop owns.
  • Random state. Where the stream that picks batches had got to.

Leave one out and the run still goes on. That is what makes it dangerous: nothing crashes. The warmup below is the schedule as the lesson's script uses it. It reads the step, and nothing else.

Quick check
A script seeds the weights' starting values but not the shuffle that picks batches. It runs twice with the same config. What differs?
  1. The order of the batches, so the two runs take different paths
  2. The starting weights, since they were already seeded
  3. The learning rate, which depends on the shuffle
  4. Nothing: the same data gives the same run
Show the answer

The order of the batches, so the two runs take different paths

Every source of randomness needs its seed. The starting weights match, but the shuffle gives each run a different batch order, so the paths and the final weights differ. A seed that covers only part of the randomness is worse than none, since it looks like control.

Quick check
What does this print: the learning rate at steps 0, 9 and 10, to three places?
import math
peak, warmup, total = 0.1, 10, 100

def lr_at(step):
    if step < warmup:
        return peak * (step + 1) / warmup
    t = (step - warmup) / (total - warmup)
    return peak * 0.5 * (1 + math.cos(math.pi * t))

print(round(lr_at(0), 3), round(lr_at(9), 3), round(lr_at(10), 3))
  1. 0.1 0.1 0.1
  2. 0.01 0.09 0.1
  3. 0.0 0.09 0.1
  4. 0.01 0.1 0.1
Show the answer

0.01 0.1 0.1

Warmup counts step + 1 of 10: step 0 gives 0.1 × 1 / 10 = 0.01, and step 9 reaches 0.1 × 10 / 10 = 0.1. At step 10 the cosine starts, at t = 0, where 0.5 × (1 + 1) = 1, so the rate is 0.1 again. The rate is a function of the step. That is why a lost step number restarts the warmup.

In your own words
In your own words: why does a run that resumed from a file holding only the weights keep going without any error, and why is that a problem?

Write a sentence or two, then compare it with a good answer in the lesson.

From scratch: stop at step 50, resume, match

A claim you can test: a run stopped at step 50 and resumed from a file should end exactly where an unbroken run of 100 steps ends. First, the idea on one weight. The cell holds a weight w, a momentum v and a step count, saves all three to bytes, reads them back as a new process would, and carries on.

Now the real run. Run in this lesson's setup block is a whole script in miniature: a config, data made from the seed, Adam written out, and a warmup then cosine schedule. It fits a line to 200 noisy points. The setup is folded above the first cell. First, the unbroken run.

Setup (runs before every cell) · 65 lines

The lesson runs this once, before its first cell, and every cell, drill and example starts with what it defines. A cell that changes a setup object changes it for the cells after it, until the page reloads.

import io
import math
import pickle
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

CONFIG = {"seed": 0, "rows": 200, "batch": 20, "lr": 0.1, "warmup": 10, "steps": 100}

def make_data(seed, rows):
    """Three inputs and a constant 1 for the bias. True weights: 1.5, -2, 0.5, 0.3."""
    rng = np.random.default_rng(seed)
    x = rng.normal(size=(rows, 3))
    y = x @ np.array([1.5, -2.0, 0.5]) + 0.3 + rng.normal(scale=0.1, size=rows)
    return np.hstack([x, np.ones((rows, 1))]), y

def lr_at(step, cfg):
    """Warm up for cfg["warmup"] steps, then fall along half a cosine to zero."""
    if step < cfg["warmup"]:
        return cfg["lr"] * (step + 1) / cfg["warmup"]
    t = (step - cfg["warmup"]) / (cfg["steps"] - cfg["warmup"])
    return cfg["lr"] * 0.5 * (1 + math.cos(math.pi * t))

class Run:
    """Linear regression trained with Adam, written out, with every part of a script."""
    def __init__(self, cfg):
        self.cfg = cfg
        self.x, self.y = make_data(cfg["seed"], cfg["rows"])
        self.rng = np.random.default_rng(cfg["seed"] + 1)       # picks each batch
        self.w, self.m, self.v = np.zeros(4), np.zeros(4), np.zeros(4)
        self.step = 0

    def loss(self):
        return float(((self.x @ self.w - self.y) ** 2).mean())

    def train(self, until):
        c = self.cfg
        while self.step < until:
            rows = self.rng.choice(c["rows"], size=c["batch"], replace=False)
            x, y = self.x[rows], self.y[rows]
            g = 2 * x.T @ (x @ self.w - y) / len(y)
            self.m = 0.9 * self.m + 0.1 * g                     # Adam's two memories
            self.v = 0.999 * self.v + 0.001 * g ** 2
            t = self.step + 1
            m_hat, v_hat = self.m / (1 - 0.9 ** t), self.v / (1 - 0.999 ** t)
            self.w = self.w - lr_at(self.step, c) * m_hat / (np.sqrt(v_hat) + 1e-8)
            self.step += 1
        return self

    def state(self):
        return {"step": self.step, "w": self.w, "m": self.m, "v": self.v,
                "rng": self.rng.bit_generator.state}

    def load(self, state, skip=()):
        """Put a saved state back. Names in skip stay as a new run has them."""
        if "step" not in skip:
            self.step = state["step"]
        if "w" not in skip:
            self.w = state["w"].copy()
        if "m" not in skip:
            self.m, self.v = state["m"].copy(), state["v"].copy()
        if "rng" not in skip:
            self.rng.bit_generator.state = state["rng"]
        return self
whole = Run(CONFIG)
print("loss at the start:", round(whole.loss(), 4))
whole.train(CONFIG["steps"])
print("loss after", whole.step, "steps:", round(whole.loss(), 4))
print("weights:", np.round(whole.w, 3), "(the true ones: 1.5, -2, 0.5, 0.3)")

Now stop at step 50 and save a checkpoint. pickle.dumps turns the state into bytes, the way torch.save does for tensors. Then build a new Run from the config, as a new process would, load the bytes, and finish. The cell runs four resumes: with everything, then each time with one part left out.

Uses the setup above, 65 lines.

whole = Run(CONFIG).train(100)

first = Run(CONFIG).train(50)
blob = pickle.dumps(first.state())
print("checkpoint is", type(blob).__name__, "holding", sorted(pickle.loads(blob)))

def resume(skip=()):
    run = Run(CONFIG)                         # a new process, built from the config
    run.load(pickle.loads(blob), skip)
    return run.train(run.step + 50)           # 50 steps are still to do

cases = [("everything", ()), ("no optimizer state", ("m",)),
         ("no random state", ("rng",)), ("no step", ("step",))]
for name, skip in cases:
    run = resume(skip)
    gap = float(np.abs(run.w - whole.w).max())
    print(name.ljust(20), "steps", run.step, "largest weight gap", f"{gap:.6f}")

With everything saved the gap is exactly 0.000000: the resumed run is the unbroken run, to every digit and not only to four places. Leave out any one part and the weights end up a little different, by 0.076 without Adam's memory, by 0.011 without the random state and by 0.070 without the step. None of them crashes. The final loss of each is close to the others, since this is a small, easy problem, so a loss curve would not show the mistake. The gap in the weights does, and only because we kept the unbroken run to compare with. A resumed run that isn't the same run is a different experiment.

The step deserves a picture. The schedule reads the step, so a lost step sends the learning rate back to the beginning of its warmup, in the middle of the run.

Uses the setup above, 65 lines.

import matplotlib.pyplot as plt

steps = range(100)
kept = [lr_at(s, CONFIG) for s in steps]
lost = kept[:50] + [lr_at(s - 50, CONFIG) for s in range(50, 100)]
plt.plot(steps, kept, label="step saved")
plt.plot(steps, lost, linestyle="--", label="step lost at 50")
plt.axvline(50, color="gray", linestyle=":")
plt.xlabel("step")
plt.ylabel("learning rate")
plt.legend()
print("rate at step 50:", round(lr_at(50, CONFIG), 4), "kept, against",
      round(lr_at(0, CONFIG), 4), "when the step is lost")
Quick check
The run is resumed and finishes with a weight gap of 0.000000 from the unbroken run. Which of these did the resume have to bring back to get that?
  1. The weights, Adam's m and v, the step and the random state
  2. The weights only: the rest is rebuilt from the config
  3. The config: it holds the learning rate and the seed
  4. The weights and the step: the optimizer starts fresh
Show the answer

The weights, Adam's m and v, the step and the random state

All four of the cell's variants that left something out ended with a gap above 0. The config is rebuilt from its own file, but it holds the starting values. The weights, m and v, the step and the random stream all changed during the run, so all of them were saved.

Quick check
What does this print: the first weight after loading the bytes, and the type of blob?
import pickle
state = {"step": 5, "w": [0.5, 1.5]}
blob = pickle.dumps(state)
state["w"][0] = 99
print(pickle.loads(blob)["w"][0], type(blob).__name__)
  1. 99 bytes
  2. 0.5 bytes
  3. 0.5 dict
  4. 99 dict
Show the answer

0.5 bytes

pickle.dumps copies the state into bytes at that moment. Changing the dictionary afterwards doesn't change the bytes, so loading them gives 0.5. That is what a checkpoint is for: a snapshot that training can't disturb.

Worked example

Worked example: what a checkpoint holds, in bytes

The run saved at step 50 holds its weights, Adam's two memories and the random state. Each of the four weights is a float64, 8 bytes. The cell counts what each part costs.

Uses the setup above, 65 lines.

run = Run(CONFIG).train(50)
state = run.state()
for key in ("w", "m", "v"):
    print(key, state[key].nbytes, "bytes")
print("model and optimizer:", sum(state[k].nbytes for k in "wmv"), "bytes")
print("the model alone:", state["w"].nbytes, "bytes")
print("the whole file:", len(pickle.dumps(state)), "bytes, with step and stream")
In the lesson you can change this and run it.

The four weights take 32 bytes, and so do m and v: three sets, 96 bytes in all, three times the model alone. The file is larger than 96, since it also holds the step, the random stream's state and the pickle's own bookkeeping. At this size none of it matters. For a model with a billion weights, Adam's two memories are the larger share of the file.

Drill

The learning rate at a step

Write lr_at(step, peak, warmup, total): for step < warmup the rate is peak * (step + 1) / warmup. From warmup on it falls along half a cosine, peak * 0.5 * (1 + cos(pi * t)), where t = (step - warmup) / (total - warmup). Use math.

Uses the setup above, 65 lines.

import math

def lr_at(step, peak, warmup, total):
    return 0.0
print(round(lr_at(0, 0.1, 10, 100), 4), round(lr_at(55, 0.1, 10, 100), 4))
In the lesson, your code is checked as you write it.
Drill

How big is a checkpoint?

Write checkpoint_bytes(n_params, optimizer, bytes_per=4). The model is n_params * bytes_per bytes. The optimizer adds more sets of the same size: "sgd" adds none, "momentum" adds 1 (the velocity), "adam" adds 2 (m and v). Raise a ValueError for any other name.

Uses the setup above, 65 lines.

def checkpoint_bytes(n_params, optimizer, bytes_per=4):
    return 0
print(checkpoint_bytes(1000, "adam"), checkpoint_bytes(1000, "sgd"))
In the lesson, your code is checked as you write it.
In your own words
In your own words: what is the difference between a model file and a checkpoint, and when would you save each?

Write a sentence or two, then compare it with a good answer in the lesson.

In PyTorch: module, dataset, checkpoint

Real scripts do all of this with objects PyTorch provides. A module, a dataset and a device run on this page. The optimizer's saved state needs real PyTorch, so it comes as a card.

A module with buffers, one device and a round trip

A buffer is a tensor the model owns that training doesn't change: a mean, a mask, a position table. register_buffer tells the module to move it with the model and save it in the checkpoint. A plain attribute is neither. nn.ModuleList holds a list of layers so that PyTorch can see them. A plain Python list hides them from parameters(), and the optimizer then never updates them.

A script also defines the device once and uses that one name everywhere: for the model, for each batch and in torch.load's map_location. This page has no GPU, so the line below picks the CPU. On a machine with one it picks the GPU. The cell saves a model and a step into an in-memory file, loads them into a new model built with the wrong mean and spread, and checks that the buffers came back with the weights.

Uses the setup above, 65 lines.

class Net(nn.Module):
    def __init__(self, mean, std, width=8, depth=2):
        super().__init__()
        self.register_buffer("mean", mean)           # saved and moved, never trained
        self.register_buffer("std", std)
        sizes = [len(mean)] + [width] * depth
        self.layers = nn.ModuleList(
            [nn.Linear(sizes[i], sizes[i + 1]) for i in range(depth)])
        self.head = nn.Linear(width, 1)

    def forward(self, x):
        x = (x - self.mean) / self.std
        for layer in self.layers:
            x = torch.relu(layer(x))
        return self.head(x).squeeze(1)

class Hidden(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = [nn.Linear(3, 3), nn.Linear(3, 3)]       # a plain list

device = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(1)
model = Net(torch.tensor([1.0, 2.0, 3.0]), torch.tensor([2.0, 2.0, 2.0])).to(device)
print("parameters:", [name for name, p in model.named_parameters()])
print("buffers:   ", [name for name, b in model.named_buffers()])
print("saved:     ", list(model.state_dict()))
print("parameters in a plain list of layers:", len(list(Hidden().parameters())))

x = torch.randn(5, 3).to(device)
file = io.BytesIO()
torch.save({"step": 50, "model": model.state_dict()}, file)
file.seek(0)
ckpt = torch.load(file, map_location=device, weights_only=True)

fresh = Net(torch.zeros(3), torch.ones(3)).to(device)       # wrong buffers on purpose
print("before loading, same output:", torch.allclose(model(x), fresh(x)))
fresh.load_state_dict(ckpt["model"])
print("after loading, same output:", torch.allclose(model(x), fresh(x)))
print("mean came back:", fresh.mean.tolist(), "| step:", ckpt["step"])

The buffers are saved but are not parameters, so the optimizer never sees them. The plain list hides all its weights: 0 parameters, so nothing would train. The fresh model gave other answers until it was loaded. Had the mean been a plain attribute, the file would not have held it, and a resumed model would have standardized its inputs with the wrong numbers, silently.

A Dataset and a collate function

A Dataset returns one example. A collate function turns a list of them into one batch. The default stacks tensors, which fails for different lengths. This one pads the short ones and returns a mask, so the loss can ignore the padding.

Uses the setup above, 65 lines.

class Tokens(Dataset):
    def __init__(self, seqs):
        self.seqs = seqs
    def __len__(self):
        return len(self.seqs)
    def __getitem__(self, i):
        return torch.tensor(self.seqs[i]), len(self.seqs[i]) % 2

def collate(batch):
    longest = max(len(ids) for ids, label in batch)
    padded = torch.zeros(len(batch), longest, dtype=torch.long)
    mask = torch.zeros(len(batch), longest, dtype=torch.bool)
    for row, (ids, label) in enumerate(batch):
        padded[row, :len(ids)] = ids
        mask[row, :len(ids)] = True
    return padded, mask, torch.tensor([label for ids, label in batch])

data = Tokens([[5, 6, 7], [8, 9], [3, 4, 5, 6, 7], [1]])
padded, mask, labels = next(iter(DataLoader(data, batch_size=4, collate_fn=collate)))
print("padded", tuple(padded.shape), "mask", tuple(mask.shape))
print("labels", labels.tolist())
print(padded.tolist())
print("real tokens per row:", mask.sum(dim=1).tolist())
Quick check
What does this print: the longest sequence, the number of real tokens, and the slots in the padded batch?
batch = [[1, 2, 3], [4], [5, 6]]
longest = max(len(ids) for ids in batch)
real = sum(len(ids) for ids in batch)
print(longest, real, longest * len(batch))
  1. 6 3 9
  2. 3 9 9
  3. 3 6 9
  4. 3 6 6
Show the answer

3 6 9

The longest has 3 tokens. There are 3 + 1 + 2 = 6 real tokens. Padding every row to 3 makes 3 × 3 = 9 slots, so 3 are padding, and the mask marks them. Sorting sequences by length before batching cuts that waste.

Quick check
What does this print: the number of parameters in the model, and the number of buffers?
import torch
import torch.nn as nn

class Scaler(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("mean", torch.zeros(3))
        self.layer = nn.Linear(3, 2)

m = Scaler()
print(len(list(m.parameters())), len(list(m.buffers())))
  1. 0 3
  2. 3 0
  3. 1 2
  4. 2 1
Show the answer

2 1

The linear layer has two parameters, its weight and its bias. The mean is one buffer. The optimizer is handed the two parameters, and the checkpoint holds all three tensors.

One more format is worth knowing by name. safetensors is a library with its own file format, which holds tensors and nothing else, so loading a file can never run code. Model hubs use it for shared weights. It stores the model's tensors only, so a training checkpoint still goes through torch.save, with the optimizer and the step beside the weights.

On the job

What goes wrong with scripts in a real week:

  • Resume without checking. Run this lesson's test: stop a short run in the middle, resume it, and compare the end with an unbroken run. It takes a minute, and it catches the missing optimizer.
  • A half-written checkpoint. A crash mid-save leaves a broken file. Write to a temporary name, then rename it over the old one. A rename is all or nothing.
  • Keep more than one. Keep the last few and the best. A run can go wrong at step 3,000 and show it at 3,200.
  • Config in the file. Save the config beside the weights, so the file says how it was made.
  • A different batch on resume. Fewer GPUs means a different batch, learning rate and steps per epoch. Check the config you resume under against the saved one.
  • Where this shows up. Debugging training reads a failing run from its numbers. A resume that quietly changed the run is the hardest to read, since every number looks plausible.

Goes deeper in: Capstone: train a seam-tear spotter, which builds the Dataset, the module, the cosine schedule and best.pt for a real task, Training at scale: precision, accumulation, data and many GPUs for accumulation and precision, and Optimizers for the memory Adam keeps.

In your own words
Your colleague's resumed run "went fine" but ended with a worse score than the unbroken one. List two things in the checkpoint you would check first, and how you would test each.

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 6 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 14 review cards for spaced repetition

Try a quiz question

A run is resumed from a file that holds only the model's weights. What happens?

  1. It runs without error, but along a different path: Adam's memory and the step start again
  2. It crashes, since the optimizer has no state to load
  3. It runs exactly as the unbroken run would have
  4. It starts training from the first step with the saved weights as a new model

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 4 · about 24 hours of study

Deep learning

Neural networks, backprop and training them well 31 lessons 1 free
  1. The artificial neuron FreeWeighted sum plus a nonlinearity.35 min
  2. Activation functionsReLU, sigmoid, GELU, and why nonlinearity matters.35 min
  3. Multi-layer networksStack layers to learn complex functions.35 min
  4. The forward passFollow numbers through a network to a prediction.36 min
  5. Computation graphsEvery calculation as a graph of small operations.40 min
  6. BackpropagationThe chain rule applied backwards to get every gradient.42 min
  7. Build an autograd engineWrite backprop yourself in ~100 lines (micrograd).42 min
  8. PyTorch tensorsNumPy-like arrays that run on GPUs.38 min
  9. PyTorch autograd and modulesLet PyTorch compute gradients; build models with nn.Module.42 min
  10. The training loopForward, loss, backward, step, repeat.40 min
  11. Mini-batches and SGDTrain on small random batches instead of everything at once.40 min
  12. OptimizersMomentum and Adam.41 min
  13. Learning-rate schedules in PyTorchWarmup then cosine by hand, as a function of the step, then LambdaLR and where scheduler.step() goes; digits trained with and without warmup.25 min
  14. Vanishing and exploding gradientsWhy deep networks were hard to train.40 min
  15. Weight initializationStart weights at the right scale.33 min
  16. Normalization layersBatch and layer norm keep training stable.40 min
  17. Residual connectionsSkip connections that let gradients flow.40 min
  18. DropoutRandomly switch off neurons to prevent overfitting.35 min
  19. Debugging trainingRead a failing run from its numbers, and fix what's wrong.40 min
  20. Looking inside a networkRead four numbers per layer (spread, dead units, gradient norm, update-to-weight ratio), collect them in NumPy, then with forward and tensor hooks, and diagnose three planted faults.30 min
  21. Transfer learningReuse a trained network: freeze the backbone, train a new head, fine-tune with two learning rates.42 min
  22. Convolutional networksPattern detectors for images.44 min
  23. Embedding layersLearned lookup tables that turn tokens into vectors.38 min
  24. Recurrent networksProcess sequences step by step, and their limits.43 min
  25. LSTMs and GRUs, built and trainedBuild an LSTM and a GRU from a linear layer, train them on the adding problem beside a plain RNN, read what the gates learned, measure the gradient's path, and carry state across a stream.40 min
  26. Sequence to sequence, and the invention of attentionBuild an encoder and decoder, measure the bottleneck of a fixed-size handoff, add attention in three lines, read an attention map with care, and see how one mistake compounds after teacher forcing.45 min
  27. AutoencodersEncoder, bottleneck and decoder, reconstruction error for anomalies, denoising, and the latent space image generators use.42 min
  28. GPUs and computeWhy deep learning runs on graphics cards.40 min
  29. Training at scale: precision, accumulation, data and many GPUsGradient accumulation and what it really saves, scaling the learning rate with the batch, mixed precision and loss scaling, a data loader that starves the GPU, and why eight GPUs aren't eight times as fast.45 min
  30. A training script, start to finishConfig, seed, data, model, schedule, loop and checkpoint as one script: what a checkpoint must hold, a resume that matches an unbroken run, and buffers, collate and torch.save in PyTorch.40 min
  31. Capstone: train a seam-tear spotterTrain and compare three models on fabric images, choose the alarm level from costs, and score once.45 min

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write