Code has had version control for decades: every change is kept, named and recoverable. Data and models mostly haven't. They sit in folders as train_final.csv and model_v2_new.pkl, too big for git, overwritten in place. Then a labeling bug turns up in a file from March, and nobody can say which of the models now serving customers were trained on it. This lesson builds the three pieces that answer that question: a store that names files by their contents, a manifest that names a whole dataset, and a lineage record that ties each model to both.
What you'll be able to do
Store a file under the hash of its contents, so a version can't change and identical content is kept once
Describe a dataset version as a manifest of file hashes, and list what changed between two versions
Give a trained model a version, and record the data and code it came from
Use that lineage to answer which models a bad file has reached
Quick check
A guess before you start. A shared folder holds train.csv, last changed yesterday. A model trained two months ago is misbehaving, and its run log says it was trained on "train.csv". What can you say about the data that model saw?
It saw no data at all, since the file is newer than the model
It saw the file as it is in the folder today, since the name matches
It saw yesterday's version, since that's the latest change on record
Not much: the name is the same, but the contents may have changed since
Show the answer
Not much: the name is the same, but the contents may have changed since
A file name is a label on a slot, and the slot's contents can be replaced. Unless something recorded what was in the file at training time, the data that model saw is gone.
The idea: a name that is the content
A version is a hash of what a thing contains. Experiment tracking used it as a fingerprint; here it becomes the name. The check below recalls what a git commit id names.
Name a file by what's in it
Experiment tracking used a hash as a fingerprint: a short code that changes whenever the content does. Take that one step further and use the hash as the file's name. This is called content addressing, and it is how git stores every version of your code.
import hashlib, os, shutil
defkey_of(data):
return hashlib.sha256(data).hexdigest()[:16]
defput(folder, data):
os.makedirs(folder, exist_ok=True)
key = key_of(data)
path = os.path.join(folder, key)
ifnot os.path.exists(path): # already stored: nothing to writewithopen(path, "wb") as f:
f.write(data)
return key
if os.path.exists("store"):
shutil.rmtree("store") # start this demonstration from an empty store
march = b"id,label\n1,spam\n2,ham\n3,spam\n"
fixed = b"id,label\n1,spam\n2,ham\n3,ham\n"# row 3 was mislabeledprint(put("store", march))
print(put("store", fixed))
print(put("store", march)) # the March file againprint(sorted(os.listdir("store")))
It runs right here in your browser, with nothing to install.
put writes the bytes to a file named after their hash and hands the name back. That name is the version. Three properties follow, and each one fixes a problem with train_final.csv:
A version can never change. Different contents would have a different name. The corrected file didn't replace the March file. It was stored beside it.
Identical content is stored once. Putting the March file a second time returned the same key and wrote nothing: the store holds two files, not three.
Damage is detectable. Hash a file you fetched and compare with its name. If they differ, the file is corrupt or has been tampered with.
Tools such as DVC and Git LFS work this way. The large file goes into a store under its hash, and a tiny pointer file holding that hash goes into git, where it is versioned along with the code that reads it.
Quick check
A content-addressed store, kept here as a dictionary from key to bytes. What does this print?
import hashlib
store = {}
defput(data):
key = hashlib.sha256(data).hexdigest()[:8]
store[key] = data
return key
a = put(b"cat")
b = put(b"dog")
c = put(b"cat")
print(len(store), a == c, a == b)
2 False False
3 True False
3 False False
2 True False
Show the answer
2 True False
The same bytes always hash to the same key, so the second b"cat" lands on the key that is already there. The store holds two entries, a and c are the same key, and b"dog" has a different one.
Try it now
Content addressing in four boxes. Git, DVC and Git LFS all work this way.
A dataset is a manifest
A real dataset is many files: training rows, test rows, a label list, perhaps thousands of images. One hash per file isn't a dataset version yet. The version is a manifest: a list of every path with the hash of its contents. Hash the manifest itself and you get one short id for the whole dataset.
import hashlib, json
defkey_of(data):
return hashlib.sha256(data).hexdigest()[:16]
defmanifest_of(files):
return {path: key_of(data) for path, data insorted(files.items())}
defversion_of(manifest):
return hashlib.sha256(
json.dumps(manifest, sort_keys=True).encode()).hexdigest()[:12]
v1 = {
"train.csv": b"id,label\n1,spam\n2,ham\n3,spam\n",
"test.csv": b"id,label\n9,ham\n",
"labels.txt": b"spam\nham\n",
}
v2 = dict(v1)
v2["train.csv"] = b"id,label\n1,spam\n2,ham\n3,ham\n"# the fix to row 3
v2["extra.csv"] = b"id,label\n4,spam\n"# a new file
m1, m2 = manifest_of(v1), manifest_of(v2)
print("v1", version_of(m1), m1)
print("v2", version_of(m2), m2)
Each dataset version is now one id: d5ac1c19e985 and 8558e408a4fa. A run record that says "data": "d5ac1c19e985" names every byte of every file, and the manifest is small enough to keep in git.
What changed between two versions falls straight out of the two manifests. Same path and same hash: unchanged. Same path, different hash: changed. A path in only one of them: added or removed.
m1 = {"labels.txt": "6139d968fbc60b5f", "test.csv": "b8d151f23126c29e",
"train.csv": "89cced1e566639dc"}
m2 = {"extra.csv": "08766600a676f603", "labels.txt": "6139d968fbc60b5f",
"test.csv": "b8d151f23126c29e", "train.csv": "9e98ff31fb424a8d"}
print("unchanged:", sorted(p for p in m1 if m2.get(p) == m1[p]))
print("changed: ", sorted(p for p in m1 if p in m2 and m2[p] != m1[p]))
print("added: ", sorted(p for p in m2 if p notin m1))
print("removed: ", sorted(p for p in m1 if p notin m2))
print("objects stored for both versions:", len(set(m1.values()) | set(m2.values())),
"| files listed:", len(m1) + len(m2))
Nobody opened a file to learn this. And because unchanged files have unchanged hashes, the store keeps five objects for the seven files the two versions list. A dataset of a million images in which a hundred changed costs a hundred images to version, not a second million.
Quick check
Two versions of a dataset, as manifests. What does this print?
old = {"a.csv": "11", "b.csv": "22", "c.csv": "33"}
new = {"a.csv": "11", "b.csv": "99", "d.csv": "44"}
changed = sorted(p for p in old if p in new and new[p] != old[p])
removed = sorted(p for p in old if p notin new)
added = sorted(p for p in new if p notin old)
print(changed, removed, added)
['b.csv'] ['d.csv'] ['c.csv']
['b.csv'] ['c.csv'] ['d.csv']
['b.csv', 'c.csv'] [] ['d.csv']
['a.csv', 'b.csv'] ['c.csv'] ['d.csv']
Show the answer
['b.csv'] ['c.csv'] ['d.csv']
b.csv is in both with a different hash, so it changed. c.csv is only in the old manifest, so it was removed. d.csv is only in the new one, so it was added. a.csv has the same hash in both and appears in no list.
In your own words
In your own words: what do you gain by naming a file with the hash of its contents?
Write a sentence or two, then compare it with a good answer in the lesson.
In code: a store, a manifest, a model file and a lineage lookup
Each cell below is a few lines of Python. DVC and Git LFS are these lines with a server and a nicer command.
Drill
A store that checks what it hands back
A store is a dictionary from key to bytes. Write put(store, data): save data under the first 16 characters of its SHA-256 hex digest and return that key. Write get(store, key): return the bytes, but first hash them again. If the hash no longer matches the key, the stored bytes have been damaged: raise ValueError. A key that isn't in the store raises KeyError.
import hashlib
defput(store, data):
# save the bytes under their own hash, and return the keyreturn""defget(store, key):
# return the bytes, after checking they still hash to the keyreturnb""
store = {}
key = put(store, b"id,label\n1,spam\n")
print(key, get(store, key))
In the lesson, your code is checked as you write it.
Drill
What changed between two dataset versions?
Write diff_manifests(old, new). Each manifest is a dictionary from path to content hash. Return a dictionary with three lists of paths, each sorted: "added" (only in new), "removed" (only in old) and "changed" (in both, with different hashes).
In the lesson, your code is checked as you write it.
Models are files too
A trained model is a file of numbers, and everything above applies to it. pickle.dumps turns a Python object, a fitted model included, into bytes, and bytes can be hashed and stored.
import hashlib, pickle
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=200, n_features=5, random_state=0)
deftrain(C):
return pickle.dumps(LogisticRegression(C=C).fit(X, y))
defkey_of(data):
return hashlib.sha256(data).hexdigest()[:16]
first, again, other = train(1.0), train(1.0), train(0.5)
print("same settings, trained twice, same key:", key_of(first) == key_of(again))
print("a different setting, same key: ", key_of(first) == key_of(other))
Train twice with the same data and settings and the two files are identical to the byte, so they share a key. Change one setting and the weights move, and so does the key. A model's hash says exactly which weights you have, which "model_v2_new.pkl" never does.
Go deeper Two cautions about pickle
A pickle file can run code when it is loaded, so never unpickle a model from a source you don't trust: that is one reason published models have moved to formats that hold only numbers, such as safetensors. And a pickle written by one version of a library may not load in another, so a model's record should say which versions it needs.
A hash tells you which model you have. It doesn't tell you where the model came from. For that, each model gets a small record written at the moment it is saved:
import json
model_record = {
"model": "21621e064a9283e6", # the hash of the model file"data": "d5ac1c19e985", # the dataset version it was trained on"code": "a41c9e7", # the git commit of the training code"params": {"C": 1.0},
"metrics": {"accuracy": 0.94},
}
print(json.dumps(model_record, indent=2))
This is the run record from the experiment tracking lesson with one more field: the hash of what the run produced. The chain it forms, from data and code through a run to a model, is called lineage.
The author who made the change and the time they made it
The single file that was edited most recently
One exact state of every tracked file in the repository
The most recent version of the main branch, whenever you look
Show the answer
One exact state of every tracked file in the repository
A commit is a snapshot of the whole tracked tree, and its id never points anywhere else. That's why a run record can name its code with a commit id, the same way it names its data with a manifest's id.
A model is also a file of numbers, so it gets a key too. With PyTorch, hash the bytes of the state_dict. This academy's torch is a teaching subset (same code, CPU only; its seeded numbers differ from real PyTorch), so the cell prints only whether two models match:
import hashlib
import torch
from torch import nn
deffingerprint(model):
blob = b"".join(t.detach().numpy().tobytes() for t in model.state_dict().values())
return hashlib.sha256(blob).hexdigest()[:12]
torch.manual_seed(0)
a = fingerprint(nn.Linear(2, 1))
torch.manual_seed(0)
b = fingerprint(nn.Linear(2, 1))
torch.manual_seed(1)
c = fingerprint(nn.Linear(2, 1))
print(a == b, a == c)
The same seed makes the same weights and the same key. A different seed gives different weights and a different key.
Following the lineage
Lineage earns its keep on a bad day. A labeling bug is found in the March training file, the one whose hash is 89cced1e566639dc. Which models are affected? With manifests and model records the answer is a lookup: find every dataset version whose manifest contains that file, then every model trained on one of those versions.
manifests = {
"d5ac1c19e985": {"labels.txt": "6139d968fbc60b5f", "test.csv": "b8d151f23126c29e",
"train.csv": "89cced1e566639dc"},
"8558e408a4fa": {"extra.csv": "08766600a676f603", "labels.txt": "6139d968fbc60b5f",
"test.csv": "b8d151f23126c29e", "train.csv": "9e98ff31fb424a8d"},
}
models = [
{"model": "m-churn-1", "data": "d5ac1c19e985", "code": "a41c9e7"},
{"model": "m-churn-2", "data": "8558e408a4fa", "code": "a41c9e7"},
{"model": "m-spam-1", "data": "d5ac1c19e985", "code": "c07be11"},
]
bad_file = "89cced1e566639dc"
bad_versions = sorted(v for v, m in manifests.items() if bad_file in m.values())
affected = [r["model"] for r in models if r["data"] in bad_versions]
print("dataset versions that include the bad file:", bad_versions)
print("models to retrain:", affected)
Two of the three models were trained on the bad file, and the third, trained after the fix, is clean. Without these records the honest answer would have been "we don't know, retrain everything".
The same lookup answers the other questions production raises. Can we rebuild this model? Its record names the data version and commit to fetch. What changed between the model that worked and the one that doesn't? Compare their records, then diff the two manifests. Someone asked for their data to be deleted: which models learned from it? Every model whose dataset version contains a file with their rows.
Versions that never change raise one question: how does anyone say "the current model"? With a name that points at a version and can be moved. That is the job of a model registry, the subject of A model registry and promotion.
Quick check
A model in production starts giving worse answers after a routine retrain. The old and new models' records show the same code commit and the same settings, but different dataset versions. What is the most direct next step?
Retrain the new model with another random seed
Roll the training code back to an earlier commit
Diff the two dataset versions' manifests
Open every data file and read it for mistakes
Show the answer
Diff the two dataset versions' manifests
The records have already ruled out the code and the settings. The data version is the one thing that differs, and two manifests can be compared in a moment to show exactly which files were added, removed or changed.
Quick check
Why keep a dataset's manifest in git, next to the training code?
So each commit pins both the code and the exact data it was written for
So the manifest's hashes are kept secret from the artifact store
So git can store the large data files themselves more compactly
So the data can be edited by hand in a pull request
Show the answer
So each commit pins both the code and the exact data it was written for
The manifest is small text, so git can version it. Checking out a commit then gives you the code and the ids of the exact data files to fetch, and the two can't drift apart.
Worked example
Worked example: from a bad file to the models that need retraining
Build the whole chain in one run: two dataset versions, a model trained on each, then the lookup that starts from one file's hash.
import hashlib, json, pickle
import numpy as np
from sklearn.linear_model import LogisticRegression
defkey_of(data):
return hashlib.sha256(data).hexdigest()[:16]
defdataset(files):
manifest = {path: key_of(data) for path, data insorted(files.items())}
version = hashlib.sha256(
json.dumps(manifest, sort_keys=True).encode()).hexdigest()[:12]
return manifest, version
rng = np.random.default_rng(0)
X = rng.normal(size=(400, 3))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
y_buggy = y.copy()
# an export bug: these rows all got the label 0
y_buggy[X[:, 0] > 0.5] = 0# fresh rows, to score both models fairly
X_new = rng.normal(size=(1000, 3))
y_new = (X_new[:, 0] + X_new[:, 1] > 0).astype(int)
manifests, records = {}, []
for name, labels in (("march", y_buggy), ("april", y)):
manifest, version = dataset({"features.npy": X.tobytes(),
"labels.npy": labels.tobytes()})
manifests[version] = manifest
model = LogisticRegression().fit(X, labels)
records.append({"name": name, "model": key_of(pickle.dumps(model)), "data": version,
"accuracy": round(float(model.score(X_new, y_new)), 3)})
print(name, "| data", version, "| accuracy on fresh rows", records[-1]["accuracy"])
bad = key_of(y_buggy.tobytes())
versions = [v for v, m in manifests.items() if bad in m.values()]
print("retrain:", [r["name"] for r in records if r["data"] in versions])
march, april = (manifests[r["data"]] for r in records)
print("files that differ:", sorted(p for p in march if march[p] != april[p]))
In the lesson you can change this and run it.
The two dataset versions share their features file and differ in the labels file, which the manifest comparison on the last line shows without opening either. The bad labels' hash appears in one manifest, that manifest's id appears in one model record, and so one model is listed for retraining. The damage is real: the model trained on the buggy labels gets 0.63 of the fresh rows right, where the one trained after the fix gets 0.993. The lookup is how you find every model in that state.
In your own words
A model's hash and a model's record answer different questions. Which is which?
Write a sentence or two, then compare it with a good answer in the lesson.
On the job: DVC pointers, hub revisions and a lineage query
On a team the store is a bucket, the manifest is a small file in git, and "which models used this file" is a query you run before anyone is paged.
Where it comes up in a week
A labeling bug is found in last month's file. You take its hash, find every dataset manifest that holds it, and every model record that names one of those datasets. The answer is a list, not a guess. For a deletion request, the same lookup says which models learned from the person's rows, and the next scheduled retrain leaves them out.
03Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
04Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
05CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
06Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
07Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
08Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
09Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
11Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
13A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
14Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
15Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
16Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
17Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
18Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
19Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
20Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
21Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
22A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
23Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
24Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
25Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
26Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
27Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
28Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
29Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
30Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
31Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
32A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
33What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
34Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
35Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
36Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
37Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
43Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.
See the course →
Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write