Ship a small machine learning library in Python
The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.
A typed, tested and profiled package with a train and an eval command, a YAML config, logging and a wheel.
This is the capstone for the whole software phase. Almost every skill in it appears here at once: a generator for the data, a typed configuration, a command line with exit codes, logging, tests that survive random numbers, a profile, and a package someone else can install. The model is deliberately tiny, a line fitted to a few points, so the engineering is what you notice. It is the same shape as the training repository you will work in: a small library with a train command, an eval command, a config file and tests.
In a config file the program reads, so runs need no edits
Settings that change between runs belong outside the code: a config file, with the values passed to the program. Then a run is a new file, not a new edit, and the settings of every run are on record.
You will build tinyfit, a small package that fits a straight line to a few points with mini-batch gradient descent. The model is simple on purpose. The package around it is the point.
It must:
train and eval commands that return exit codes;This brings together Command-line programs (the commands), Logging (the log lines), Config files: YAML and TOML (the YAML), Testing numeric and random code (tests and seeds), Profiling numeric code and a training loop (the profile) and Packaging your code (the wheel).
The plan. Each box becomes a function or a file, and each has its own test.
Write a sentence or two, then compare it with a good answer in the lesson.
Build the library in four pieces, and check each before the next. The data first, since everything uses it.
A generator yields one batch at a time. It keeps no list of all the batches, so it works the same on ten points or ten million, and the training loop reads from it with a plain for:
import inspect def batches(xs, ys, size): for start in range(0, len(xs), size): yield xs[start:start + size], ys[start:start + size] xs = list(range(10)) ys = [2 * x + 1 for x in xs] stream = batches(xs, ys, 4) print(inspect.isgenerator(stream), next(stream)) print([len(bx) for bx, by in batches(xs, ys, 4)])
Ten points in batches of four make batches of 4, 4 and 2: the last one is smaller, and nothing is dropped.
def batches(xs, size): for start in range(0, len(xs), size): yield xs[start:start + size] print([len(b) for b in batches(list(range(10)), 3)])
[3, 3, 3, 1]
Starts 0, 3, 6 and 9 give three batches of 3 and a last one of 1. Slicing past the end just returns what is there, so no point is lost.
main(argv) returns an exit code. What should tinyfit eval return when the fitted line is not good enough?A non-zero code, such as 1
Scripts and CI read only the exit code, so a failure must return non-zero. The words printed are for people. This is the convention the whole command-line lesson builds on.
The settings go in a YAML file and come out as a dataclass, so a misspelled setting is an error and a wrong type is converted on the way in. There is one trap: PyYAML reads a number written 3e-4 as the text "3e-4", because its rule for floats wants a dot. A model trains with a learning rate of text, and the error appears much later. Writing 3.0e-4 works, and converting with float makes the loader safe either way:
from dataclasses import dataclass import yaml @dataclass class Config: lr: float = 0.05 epochs: int = 40 batch: int = 4 seed: int = 0 def load_config(text): raw = yaml.safe_load(text) or {} types = {"lr": float, "epochs": int, "batch": int, "seed": int} for key in raw: if key not in types: raise ValueError(f"unknown setting: {key}") return Config(**{key: types[key](value) for key, value in raw.items()}) print(yaml.safe_load("lr: 3e-4")) print(load_config("lr: 3e-4\nepochs: 5")) try: load_config("lrr: 0.1") except ValueError as error: print(error)
The first line shows the trap: a string. The loader's float(value) turns it into 0.0003, and a typo like lrr is refused by name.
lr: 3e-4 as the text '3e-4'. What is the safe fix in the loader?Convert each value to its type, such as float(value)
A typed config turns each setting into the type it should have when it is loaded. float('3e-4') is 0.0003, and an int field gets int(value). A wrong type then fails at load time, not in the middle of a run.
import yaml print(yaml.safe_load("epochs: 5\nname: run1"))
{'epochs': 5, 'name': 'run1'}
YAML reads a plain whole number as an int and a bare word as a string, and gives a dict. Only a float written without a dot, like 3e-4, is read as text.
Write load_config(text): read the YAML text, return a dict with lr as a float and epochs as an int, and raise ValueError(f"unknown setting: {key}") for any other key. Missing keys are fine.
import yaml def load_config(text): return yaml.safe_load(text) print(load_config("lr: 3e-4\nepochs: 5"))
Write batches(xs, size) as a generator: it yields xs[0:size], then the next size items, and so on, ending with a shorter batch if needed. It must not build a list of all the batches first.
def batches(xs, size): return [xs] print([len(b) for b in batches(list(range(10)), 4)])
Three tests are enough to guard the library. The shuffle takes its generator from the config's seed, so every test repeats. One test trains a model on purpose and expects the loss to fall:
import random def batches(xs, ys, size): for start in range(0, len(xs), size): yield xs[start:start + size], ys[start:start + size] def train(lr, epochs, batch, seed, xs, ys): rng = random.Random(seed) w = b = 0.0 order = list(range(len(xs))) losses = [] for epoch in range(epochs): rng.shuffle(order) sx, sy = [xs[i] for i in order], [ys[i] for i in order] for bx, by in batches(sx, sy, batch): errs = [w * x + b - y for x, y in zip(bx, by)] w -= lr * 2 * sum(e * x for e, x in zip(errs, bx)) / len(bx) b -= lr * 2 * sum(errs) / len(bx) losses.append(sum((w * x + b - y) ** 2 for x, y in zip(xs, ys)) / len(xs)) return w, b, losses xs = [x / 10 for x in range(10)] ys = [2 * x + 1 for x in xs] def test_loss_falls(): *_, losses = train(0.1, 300, 4, 0, xs, ys) assert losses[-1] < losses[0] / 100 def test_repeats_with_a_seed(): assert train(0.1, 10, 4, 0, xs, ys) == train(0.1, 10, 4, 0, xs, ys) def test_finds_the_line(): w, b, _ = train(0.1, 500, 4, 0, xs, ys) assert abs(w - 2) < 0.05 and abs(b - 1) < 0.05 for test in (test_loss_falls, test_repeats_with_a_seed, test_finds_the_line): test() print(test.__name__, "passed")
The loss test is the "overfit one batch" idea: a model that cannot fit ten points on a line is broken. The repeat test is only possible because the seed goes in. The last test checks the answer against what the data was made from, with a tolerance.
The training loop above does the update with a Python loop over each batch. Profile it, find the line that is called most, and fix just that. cProfile counts calls, and counts do not change between runs. The fix keeps the answer the same, which the cell checks:
import cProfile import pstats import random import numpy as np def predict(w, b, x): return w * x + b def batches(xs, ys, size): for start in range(0, len(xs), size): yield xs[start:start + size], ys[start:start + size] def fit_loop(xs, ys, epochs=20, lr=0.1, size=4): rng, w, b = random.Random(0), 0.0, 0.0 order = list(range(len(xs))) for _ in range(epochs): rng.shuffle(order) for bx, by in batches([xs[i] for i in order], [ys[i] for i in order], size): gw = gb = 0.0 for x, y in zip(bx, by): err = predict(w, b, x) - y gw, gb = gw + err * x, gb + err w -= lr * 2 * gw / len(bx) b -= lr * 2 * gb / len(bx) return w, b def fit_array(xs, ys, epochs=20, lr=0.1, size=4): rng, w, b = random.Random(0), 0.0, 0.0 order = list(range(len(xs))) for _ in range(epochs): rng.shuffle(order) for bx, by in batches([xs[i] for i in order], [ys[i] for i in order], size): bx, by = np.array(bx), np.array(by) err = predict(w, b, bx) - by w -= lr * 2 * (err * bx).mean() b -= lr * 2 * err.mean() return w, b def calls_to_predict(fit, xs, ys): profiler = cProfile.Profile() result = profiler.runcall(fit, xs, ys) stats = pstats.Stats(profiler).stats return result, {func[2]: row[1] for func, row in stats.items()}["predict"] xs = [x / 10 for x in range(10)] ys = [2 * x + 1 for x in xs] (before, calls_before), (after, calls_after) = (calls_to_predict(f, xs, ys) for f in (fit_loop, fit_array)) print(calls_before, calls_after, np.allclose(before, after))
The loop called predict once per point per epoch, 200 times. The array version calls it once per batch, 60 times, and the answer is the same to within rounding. A measured result like that, a call count before and after with a proof that nothing changed, is what goes in the pull request.
predict before the fix and 60 after. 10 points, 20 epochs, batches of 4. Why 60?3 batches per epoch, 20 epochs: one call per batch
Ten points in batches of four make 3 batches per epoch (4, 4 and 2), and 3 × 20 epochs is 60 calls, one for each batch. Before the fix, each of the 10 points had its own call: 10 × 20 = 200.
Write a sentence or two, then compare it with a good answer in the lesson.
A library is finished when someone else can install it, run it and read what it does.
main(argv) takes the words, runs a subcommand and returns an exit code, as in Command-line programs. The cell trains on the built-in points, prints one JSON line per epoch for the last two epochs, and returns 0 only when the fitted line is close to the truth:
import argparse import json import random def fit(lr, epochs, seed): xs = [x / 10 for x in range(10)] ys = [2 * x + 1 for x in xs] rng, w, b = random.Random(seed), 0.0, 0.0 order = list(range(10)) rows = [] for epoch in range(1, epochs + 1): rng.shuffle(order) for i in order: err = w * xs[i] + b - ys[i] w -= lr * 2 * err * xs[i] b -= lr * 2 * err loss = sum((w * x + b - y) ** 2 for x, y in zip(xs, ys)) / 10 rows.append({"epoch": epoch, "loss": round(loss, 6)}) return w, b, rows def main(argv): parser = argparse.ArgumentParser(prog="tinyfit") sub = parser.add_subparsers(dest="command", required=True) for name in ("train", "eval"): p = sub.add_parser(name) p.add_argument("--lr", type=float, default=0.05) p.add_argument("--epochs", type=int, default=200) p.add_argument("--seed", type=int, default=0) args = parser.parse_args(argv) w, b, rows = fit(args.lr, args.epochs, args.seed) if args.command == "train": for row in rows[-2:]: print(json.dumps(row)) return 0 good = abs(w - 2) < 0.1 and abs(b - 1) < 0.1 print("ok" if good else "not close enough") return 0 if good else 1 print(main(["train", "--epochs", "200"])) print(main(["eval", "--epochs", "200"])) print(main(["eval", "--epochs", "2"]))
train prints its last two log lines and returns 0. eval returns 0 when the fitted line is near the truth and 1 when it is not, so a CI step or a shell script can read the result without parsing anything.
A wheel is a zip file with the package's files and a small .dist-info folder that describes them. Build one by hand, read its metadata back and list what is inside, as an installer would.
import io import zipfile buf = io.BytesIO() with zipfile.ZipFile(buf, "w") as wheel: wheel.writestr("tinyfit/__init__.py", "__version__ = '0.1.0'\n") wheel.writestr("tinyfit-0.1.0.dist-info/METADATA", "Metadata-Version: 2.1\nName: tinyfit\nVersion: 0.1.0\n") wheel.writestr("tinyfit-0.1.0.dist-info/WHEEL", "Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py3-none-any\n") with zipfile.ZipFile(io.BytesIO(buf.getvalue())) as wheel: print(wheel.namelist()) meta = wheel.read("tinyfit-0.1.0.dist-info/METADATA").decode() print(meta.splitlines()[1:])
The file name tinyfit-0.1.0-py3-none-any.whl encodes the name, the version and that it works with any Python and any platform. Inside are the package folder and a .dist-info folder whose METADATA says what the package is. Real builds are made by python -m build, from the pyproject.toml, which writes these same files and adds a RECORD of every file's hash. An installer unzips the wheel into the environment. Nothing runs at install time.
A zip of the package's files plus a .dist-info folder
A pure-Python wheel is a zip. An installer reads its METADATA, checks the dependencies and unzips the files into the environment.
Build and ship your own version of tinyfit, or a small library of your choosing that fits a model to data. Treat it as a portfolio piece: someone reading your repository should be able to install it, run it and see that you tested and measured it.
Write the README first: what the library does, how to install it and the two commands to run.
import random from dataclasses import dataclass @dataclass class Config: lr: float = 0.05 epochs: int = 40 seed: int = 0 def batches(xs, ys, size): ... def train(cfg, xs, ys): rng = random.Random(cfg.seed) ... # your code: load_config, evaluate, main(argv), tests, and a profile
On the job: a library like this is what a teammate asks for in a pull request: the change, a test that fails without it, and a measured result. Put the config in the repository, pin the versions that matter, and make the exit code the truth. Then a failed run is a red CI job, not a surprise at the end of the week.
tinyfit train, and it works on your data but fails on theirs with a confusing error. What in the library would you look at first, and what would you add?Write a sentence or two, then compare it with a good answer in the lesson.
python -m build writes.Why does the dataset iterator use yield instead of returning a list of every batch?
575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.
The core of a typed, tested training package: a config that rejects unknown settings, mini-batches as a generator, a seeded training loop and an evaluation function.
Go from your first line of code to working as a Python developer: projects, testing, data, databases, web apps and APIs, all in your browser.
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write