Index Ai Academy
Phase 11 · ML in production

Versioning data and models

Files stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.

About 35 minutes · A free account opens the whole lesson.
Already have an account? Sign in to open this lesson

What it covers6

Why it matters

Code has had version control for decades: every change is kept, named and recoverable. Data and models mostly haven't. They sit in folders as train_final.csv and model_v2_new.pkl, too big for git, overwritten in place. Then a labeling bug turns up in a file from March, and nobody can say which of the models now serving customers were trained on it. This lesson builds the three pieces that answer that question: a store that names files by their contents, a manifest that names a whole dataset, and a lineage record that ties each model to both.

What you'll be able to do

  • Store a file under the hash of its contents, so a version can't change and identical content is kept once
  • Describe a dataset version as a manifest of file hashes, and list what changed between two versions
  • Give a trained model a version, and record the data and code it came from
  • Use that lineage to answer which models a bad file has reached
Quick check
A guess before you start. A shared folder holds train.csv, last changed yesterday. A model trained two months ago is misbehaving, and its run log says it was trained on "train.csv". What can you say about the data that model saw?
  1. It saw no data at all, since the file is newer than the model
  2. It saw the file as it is in the folder today, since the name matches
  3. It saw yesterday's version, since that's the latest change on record
  4. Not much: the name is the same, but the contents may have changed since
Show the answer

Not much: the name is the same, but the contents may have changed since

A file name is a label on a slot, and the slot's contents can be replaced. Unless something recorded what was in the file at training time, the data that model saw is gone.

The idea: a name that is the content

A version is a hash of what a thing contains. Experiment tracking used it as a fingerprint; here it becomes the name. The check below recalls what a git commit id names.

Name a file by what's in it

Experiment tracking used a hash as a fingerprint: a short code that changes whenever the content does. Take that one step further and use the hash as the file's name. This is called content addressing, and it is how git stores every version of your code.

import hashlib, os, shutil

def key_of(data):
    return hashlib.sha256(data).hexdigest()[:16]

def put(folder, data):
    os.makedirs(folder, exist_ok=True)
    key = key_of(data)
    path = os.path.join(folder, key)
    if not os.path.exists(path):          # already stored: nothing to write
        with open(path, "wb") as f:
            f.write(data)
    return key

if os.path.exists("store"):
    shutil.rmtree("store")                # start this demonstration from an empty store

march = b"id,label\n1,spam\n2,ham\n3,spam\n"
fixed = b"id,label\n1,spam\n2,ham\n3,ham\n"     # row 3 was mislabeled
print(put("store", march))
print(put("store", fixed))
print(put("store", march))                # the March file again
print(sorted(os.listdir("store")))
It runs right here in your browser, with nothing to install.

put writes the bytes to a file named after their hash and hands the name back. That name is the version. Three properties follow, and each one fixes a problem with train_final.csv:

  • A version can never change. Different contents would have a different name. The corrected file didn't replace the March file. It was stored beside it.
  • Identical content is stored once. Putting the March file a second time returned the same key and wrote nothing: the store holds two files, not three.
  • Damage is detectable. Hash a file you fetched and compare with its name. If they differ, the file is corrupt or has been tampered with.

Tools such as DVC and Git LFS work this way. The large file goes into a store under its hash, and a tiny pointer file holding that hash goes into git, where it is versioned along with the code that reads it.

Quick check
A content-addressed store, kept here as a dictionary from key to bytes. What does this print?
import hashlib

store = {}
def put(data):
    key = hashlib.sha256(data).hexdigest()[:8]
    store[key] = data
    return key

a = put(b"cat")
b = put(b"dog")
c = put(b"cat")
print(len(store), a == c, a == b)
  1. 2 False False
  2. 3 True False
  3. 3 False False
  4. 2 True False
Show the answer

2 True False

The same bytes always hash to the same key, so the second b"cat" lands on the key that is already there. The store holds two entries, a and c are the same key, and b"dog" has a different one.

Try it now

Content addressing in four boxes. Git, DVC and Git LFS all work this way.

A dataset is a manifest

A real dataset is many files: training rows, test rows, a label list, perhaps thousands of images. One hash per file isn't a dataset version yet. The version is a manifest: a list of every path with the hash of its contents. Hash the manifest itself and you get one short id for the whole dataset.

import hashlib, json

def key_of(data):
    return hashlib.sha256(data).hexdigest()[:16]

def manifest_of(files):
    return {path: key_of(data) for path, data in sorted(files.items())}

def version_of(manifest):
    return hashlib.sha256(
        json.dumps(manifest, sort_keys=True).encode()).hexdigest()[:12]

v1 = {
    "train.csv": b"id,label\n1,spam\n2,ham\n3,spam\n",
    "test.csv": b"id,label\n9,ham\n",
    "labels.txt": b"spam\nham\n",
}
v2 = dict(v1)
v2["train.csv"] = b"id,label\n1,spam\n2,ham\n3,ham\n"      # the fix to row 3
v2["extra.csv"] = b"id,label\n4,spam\n"                    # a new file

m1, m2 = manifest_of(v1), manifest_of(v2)
print("v1", version_of(m1), m1)
print("v2", version_of(m2), m2)

Each dataset version is now one id: d5ac1c19e985 and 8558e408a4fa. A run record that says "data": "d5ac1c19e985" names every byte of every file, and the manifest is small enough to keep in git.

What changed between two versions falls straight out of the two manifests. Same path and same hash: unchanged. Same path, different hash: changed. A path in only one of them: added or removed.

m1 = {"labels.txt": "6139d968fbc60b5f", "test.csv": "b8d151f23126c29e",
      "train.csv": "89cced1e566639dc"}
m2 = {"extra.csv": "08766600a676f603", "labels.txt": "6139d968fbc60b5f",
      "test.csv": "b8d151f23126c29e", "train.csv": "9e98ff31fb424a8d"}

print("unchanged:", sorted(p for p in m1 if m2.get(p) == m1[p]))
print("changed:  ", sorted(p for p in m1 if p in m2 and m2[p] != m1[p]))
print("added:    ", sorted(p for p in m2 if p not in m1))
print("removed:  ", sorted(p for p in m1 if p not in m2))
print("objects stored for both versions:", len(set(m1.values()) | set(m2.values())),
      "| files listed:", len(m1) + len(m2))

Nobody opened a file to learn this. And because unchanged files have unchanged hashes, the store keeps five objects for the seven files the two versions list. A dataset of a million images in which a hundred changed costs a hundred images to version, not a second million.

Quick check
Two versions of a dataset, as manifests. What does this print?
old = {"a.csv": "11", "b.csv": "22", "c.csv": "33"}
new = {"a.csv": "11", "b.csv": "99", "d.csv": "44"}
changed = sorted(p for p in old if p in new and new[p] != old[p])
removed = sorted(p for p in old if p not in new)
added = sorted(p for p in new if p not in old)
print(changed, removed, added)
  1. ['b.csv'] ['d.csv'] ['c.csv']
  2. ['b.csv'] ['c.csv'] ['d.csv']
  3. ['b.csv', 'c.csv'] [] ['d.csv']
  4. ['a.csv', 'b.csv'] ['c.csv'] ['d.csv']
Show the answer

['b.csv'] ['c.csv'] ['d.csv']

b.csv is in both with a different hash, so it changed. c.csv is only in the old manifest, so it was removed. d.csv is only in the new one, so it was added. a.csv has the same hash in both and appears in no list.

In your own words
In your own words: what do you gain by naming a file with the hash of its contents?

Write a sentence or two, then compare it with a good answer in the lesson.

In code: a store, a manifest, a model file and a lineage lookup

Each cell below is a few lines of Python. DVC and Git LFS are these lines with a server and a nicer command.

Drill

A store that checks what it hands back

A store is a dictionary from key to bytes. Write put(store, data): save data under the first 16 characters of its SHA-256 hex digest and return that key. Write get(store, key): return the bytes, but first hash them again. If the hash no longer matches the key, the stored bytes have been damaged: raise ValueError. A key that isn't in the store raises KeyError.

import hashlib


def put(store, data):
    # save the bytes under their own hash, and return the key
    return ""


def get(store, key):
    # return the bytes, after checking they still hash to the key
    return b""


store = {}
key = put(store, b"id,label\n1,spam\n")
print(key, get(store, key))
In the lesson, your code is checked as you write it.
Drill

What changed between two dataset versions?

Write diff_manifests(old, new). Each manifest is a dictionary from path to content hash. Return a dictionary with three lists of paths, each sorted: "added" (only in new), "removed" (only in old) and "changed" (in both, with different hashes).

def diff_manifests(old, new):
    return {"added": [], "removed": [], "changed": []}


old = {"train.csv": "89cc", "test.csv": "b8d1"}
new = {"train.csv": "9e98", "test.csv": "b8d1", "extra.csv": "0876"}
print(diff_manifests(old, new))
In the lesson, your code is checked as you write it.

Models are files too

A trained model is a file of numbers, and everything above applies to it. pickle.dumps turns a Python object, a fitted model included, into bytes, and bytes can be hashed and stored.

import hashlib, pickle
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=200, n_features=5, random_state=0)

def train(C):
    return pickle.dumps(LogisticRegression(C=C).fit(X, y))

def key_of(data):
    return hashlib.sha256(data).hexdigest()[:16]

first, again, other = train(1.0), train(1.0), train(0.5)
print("same settings, trained twice, same key:", key_of(first) == key_of(again))
print("a different setting, same key:         ", key_of(first) == key_of(other))

Train twice with the same data and settings and the two files are identical to the byte, so they share a key. Change one setting and the weights move, and so does the key. A model's hash says exactly which weights you have, which "model_v2_new.pkl" never does.

Go deeper Two cautions about pickle

A pickle file can run code when it is loaded, so never unpickle a model from a source you don't trust: that is one reason published models have moved to formats that hold only numbers, such as safetensors. And a pickle written by one version of a library may not load in another, so a model's record should say which versions it needs.

A hash tells you which model you have. It doesn't tell you where the model came from. For that, each model gets a small record written at the moment it is saved:

import json

model_record = {
    "model": "21621e064a9283e6",          # the hash of the model file
    "data": "d5ac1c19e985",               # the dataset version it was trained on
    "code": "a41c9e7",                    # the git commit of the training code
    "params": {"C": 1.0},
    "metrics": {"accuracy": 0.94},
}
print(json.dumps(model_record, indent=2))

This is the run record from the experiment tracking lesson with one more field: the hash of what the run produced. The chain it forms, from data and code through a run to a model, is called lineage.

Quick check
From Git and version control: what does a commit id such as a41c9e7 identify?
  1. The author who made the change and the time they made it
  2. The single file that was edited most recently
  3. One exact state of every tracked file in the repository
  4. The most recent version of the main branch, whenever you look
Show the answer

One exact state of every tracked file in the repository

A commit is a snapshot of the whole tracked tree, and its id never points anywhere else. That's why a run record can name its code with a commit id, the same way it names its data with a manifest's id.

A model is also a file of numbers, so it gets a key too. With PyTorch, hash the bytes of the state_dict. This academy's torch is a teaching subset (same code, CPU only; its seeded numbers differ from real PyTorch), so the cell prints only whether two models match:

import hashlib
import torch
from torch import nn


def fingerprint(model):
    blob = b"".join(t.detach().numpy().tobytes() for t in model.state_dict().values())
    return hashlib.sha256(blob).hexdigest()[:12]


torch.manual_seed(0)
a = fingerprint(nn.Linear(2, 1))
torch.manual_seed(0)
b = fingerprint(nn.Linear(2, 1))
torch.manual_seed(1)
c = fingerprint(nn.Linear(2, 1))
print(a == b, a == c)

The same seed makes the same weights and the same key. A different seed gives different weights and a different key.

Following the lineage

Lineage earns its keep on a bad day. A labeling bug is found in the March training file, the one whose hash is 89cced1e566639dc. Which models are affected? With manifests and model records the answer is a lookup: find every dataset version whose manifest contains that file, then every model trained on one of those versions.

manifests = {
    "d5ac1c19e985": {"labels.txt": "6139d968fbc60b5f", "test.csv": "b8d151f23126c29e",
                     "train.csv": "89cced1e566639dc"},
    "8558e408a4fa": {"extra.csv": "08766600a676f603", "labels.txt": "6139d968fbc60b5f",
                     "test.csv": "b8d151f23126c29e", "train.csv": "9e98ff31fb424a8d"},
}
models = [
    {"model": "m-churn-1", "data": "d5ac1c19e985", "code": "a41c9e7"},
    {"model": "m-churn-2", "data": "8558e408a4fa", "code": "a41c9e7"},
    {"model": "m-spam-1", "data": "d5ac1c19e985", "code": "c07be11"},
]

bad_file = "89cced1e566639dc"
bad_versions = sorted(v for v, m in manifests.items() if bad_file in m.values())
affected = [r["model"] for r in models if r["data"] in bad_versions]
print("dataset versions that include the bad file:", bad_versions)
print("models to retrain:", affected)

Two of the three models were trained on the bad file, and the third, trained after the fix, is clean. Without these records the honest answer would have been "we don't know, retrain everything".

The same lookup answers the other questions production raises. Can we rebuild this model? Its record names the data version and commit to fetch. What changed between the model that worked and the one that doesn't? Compare their records, then diff the two manifests. Someone asked for their data to be deleted: which models learned from it? Every model whose dataset version contains a file with their rows.

Versions that never change raise one question: how does anyone say "the current model"? With a name that points at a version and can be moved. That is the job of a model registry, the subject of A model registry and promotion.

Quick check
A model in production starts giving worse answers after a routine retrain. The old and new models' records show the same code commit and the same settings, but different dataset versions. What is the most direct next step?
  1. Retrain the new model with another random seed
  2. Roll the training code back to an earlier commit
  3. Diff the two dataset versions' manifests
  4. Open every data file and read it for mistakes
Show the answer

Diff the two dataset versions' manifests

The records have already ruled out the code and the settings. The data version is the one thing that differs, and two manifests can be compared in a moment to show exactly which files were added, removed or changed.

Quick check
Why keep a dataset's manifest in git, next to the training code?
  1. So each commit pins both the code and the exact data it was written for
  2. So the manifest's hashes are kept secret from the artifact store
  3. So git can store the large data files themselves more compactly
  4. So the data can be edited by hand in a pull request
Show the answer

So each commit pins both the code and the exact data it was written for

The manifest is small text, so git can version it. Checking out a commit then gives you the code and the ids of the exact data files to fetch, and the two can't drift apart.

Worked example

Worked example: from a bad file to the models that need retraining

Build the whole chain in one run: two dataset versions, a model trained on each, then the lookup that starts from one file's hash.

import hashlib, json, pickle
import numpy as np
from sklearn.linear_model import LogisticRegression

def key_of(data):
    return hashlib.sha256(data).hexdigest()[:16]

def dataset(files):
    manifest = {path: key_of(data) for path, data in sorted(files.items())}
    version = hashlib.sha256(
        json.dumps(manifest, sort_keys=True).encode()).hexdigest()[:12]
    return manifest, version

rng = np.random.default_rng(0)
X = rng.normal(size=(400, 3))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
y_buggy = y.copy()
# an export bug: these rows all got the label 0
y_buggy[X[:, 0] > 0.5] = 0
# fresh rows, to score both models fairly
X_new = rng.normal(size=(1000, 3))
y_new = (X_new[:, 0] + X_new[:, 1] > 0).astype(int)

manifests, records = {}, []
for name, labels in (("march", y_buggy), ("april", y)):
    manifest, version = dataset({"features.npy": X.tobytes(),
                                 "labels.npy": labels.tobytes()})
    manifests[version] = manifest
    model = LogisticRegression().fit(X, labels)
    records.append({"name": name, "model": key_of(pickle.dumps(model)), "data": version,
                    "accuracy": round(float(model.score(X_new, y_new)), 3)})
    print(name, "| data", version, "| accuracy on fresh rows", records[-1]["accuracy"])

bad = key_of(y_buggy.tobytes())
versions = [v for v, m in manifests.items() if bad in m.values()]
print("retrain:", [r["name"] for r in records if r["data"] in versions])
march, april = (manifests[r["data"]] for r in records)
print("files that differ:", sorted(p for p in march if march[p] != april[p]))
In the lesson you can change this and run it.

The two dataset versions share their features file and differ in the labels file, which the manifest comparison on the last line shows without opening either. The bad labels' hash appears in one manifest, that manifest's id appears in one model record, and so one model is listed for retraining. The damage is real: the model trained on the buggy labels gets 0.63 of the fresh rows right, where the one trained after the fix gets 0.993. The lookup is how you find every model in that state.

In your own words
A model's hash and a model's record answer different questions. Which is which?

Write a sentence or two, then compare it with a good answer in the lesson.

On the job: DVC pointers, hub revisions and a lineage query

On a team the store is a bucket, the manifest is a small file in git, and "which models used this file" is a query you run before anyone is paged.

Where it comes up in a week

A labeling bug is found in last month's file. You take its hash, find every dataset manifest that holds it, and every model record that names one of those datasets. The answer is a list, not a guess. For a deletion request, the same lookup says which models learned from the person's rows, and the next scheduled retrain leaves them out.

Goes deeper in

In your own words
What is a manifest, and why can one short id stand for a whole dataset?

Write a sentence or two, then compare it with a good answer in the lesson.

Practice what you just read

  • 8 code examples you can edit and run
  • a practice exercise that checks your code
  • a 5-question quiz
  • 11 review cards for spaced repetition

Try a quiz question

What does a content-addressed store use as a file's name?

  1. The hash of the file's contents
  2. The file's original name, with a date added
  3. A counter that goes up by one for each file
  4. The name of the person who uploaded it

Answer it at the end of the lesson to see if you're right, and why.

Before this lesson

Keep learning

Where this lesson leads.

575 lessons in thirteen phases. 176 are free without an account, and a free account opens the rest. See every lesson.

Phase 11 · about 34 hours of study

Professional practice

Design, ship and run real systems, work well in a team, and get hired 43 lessons 1 free
  1. System design fundamentals FreeRequirements, rough numbers, the standard parts and where systems break.40 min
  2. Caches, queues and background jobsSpeed reads with caches, and move slow work to queues that retry safely.40 min
  3. Designing AI systemsLatency, cost, routing, fallbacks and guardrails for features built on models.40 min
  4. Containers and deploymentPackage an app as an image, configure it with the environment, and roll out safely.45 min
  5. CI/CD pipelinesTest every change automatically, and ship with gates, flags and rollbacks.40 min
  6. Docker ComposeSeveral containers together: service names, volumes, secrets and start order.40 min
  7. Kubernetes basicsDesired state, Deployments and Services, labels, probes and rolling updates.40 min
  8. Cloud for AI engineersAccess policies and least privilege, storage, compute choices, managed model endpoints, secrets and budgets.30 min
  9. Metrics, dashboards and alert rulesCounters, gauges and histograms, a rate from a counter, a percentile from buckets, a dashboard panel and an alert rule that fires on a sustained condition.40 min
  10. Reliability and incidentsSLOs, error budgets, alerts, on-call and blameless postmortems.40 min
  11. Incident lab: from the page to the postmortemOne incident run from the page to the all-clear: decide the mitigation, work out the impact in failed requests and error budget, and write a blameless postmortem, a status update and a runbook entry.40 min
  12. Measuring an AI feature from its logsTyped views over event logs, metric specs, cost from a price table, and rollups you can rerun.40 min
  13. A/B testing a model changeHash assignment, per-user results from raw logs, sample ratio checks, guardrails and a decision rule.40 min
  14. Columnar files and a local warehouseRows against columns, the encodings that shrink a column, Parquet's row groups and the ranges that let a query skip them, and SQL over files with DuckDB.40 min
  15. Data quality checks and contractsSix properties of a delivery you can test, each as a count of rows that break it, a contract with the producer, and when to block, quarantine or warn.40 min
  16. Transformations: staging, models and testsRaw, staging, facts and marts as SQL models run in dependency order in DuckDB, tests as queries that count the rows that break a rule, an incremental load that is safe to rerun, and the same models as dbt files.45 min
  17. Streams and windowsEvents that never stop arriving: event time against arrival time, tumbling, sliding and session windows, a watermark for late events, and counting each event once.40 min
  18. Labels and annotator agreementLabels as measurements: agreement and Cohen's kappa, the pair of classes annotators confuse, votes and gold questions, and what wrong labels do to a model and a test score.45 min
  19. Experiment tracking and reproducible runsA run record of settings, seed, data and code behind every score, a run log in a dozen lines, and comparing settings across seeds.40 min
  20. Hyperparameter searchWhat a grid costs, random search on a log scale, stopping weak settings early, and why the winner's validation score flatters it.40 min
  21. Versioning data and modelsFiles stored under the hash of their contents, a manifest that names a whole dataset, and lineage from data and code to each model.35 min
  22. A model registry and promotionVersions under one name, an alias for the live one, gates a candidate must pass, group by group, and a rollback that moves a pointer.40 min
  23. Feature pipelines and training-serving skewWhy a model that passes offline fails live: statistics refit at serving, two code paths, and features from the future, with the fix for each.40 min
  24. Batch and online inferencePredict ahead of time or on request: when the inputs exist, latency near a server's limit, batching what waits, cost, and a hybrid with a freshness limit.40 min
  25. Monitoring a model: drift and decayWhy a model worsens with no code change: data drift and concept drift, a measure of how far inputs have moved, thresholds set from noise, and labels that arrive late.45 min
  26. Retraining and pipelinesWhen to retrain and on which weeks, training as steps in dependency order, a cache that reruns only what changed, and guards before and after.45 min
  27. Designing ML systemsOne method for an ML design, goal and metric to retraining, with the sums that size it: requests a second at peak, a year of feature storage, a label delay and a latency budget.45 min
  28. Datasheets: documenting a datasetThe document that travels with a dataset: a profile by column and by group, claims a program can check, splits that don't leak, a scan for personal data, and the license and limits only people can write.45 min
  29. Curation, active learning and synthetic dataThree levers on a training set: removing copies and broken rows, letting a model choose which rows to label, and manufacturing rows, which helps only when their source knows more than you do.45 min
  30. Measuring fairnessBreak a model's results down by group: selection rate, true and false positive rates and precision, why they can't all be equal, what each fix costs, and when a group is too small to judge.40 min
  31. Privacy engineeringWhy removing names isn't anonymity, how to hold less (minimization, retention, keyed pseudonyms), the idea behind differential privacy and its budget, what a model remembers, and a deletion that reaches every store.40 min
  32. A red-team labAttack a simulated assistant methodically: eight kinds of attack run as a suite, success rates beside ordinary requests blocked, fresh attacks to test the fixes, where a defense belongs, and findings kept as release tests.45 min
  33. What regulation asks of an engineerRisk tiers that follow a system's use, the requirements for high-risk systems as artifacts an engineer builds, evidence tied to a model version, a model card a program can check, and legal dates kept as data.35 min
  34. Git for teamsBranches, pull requests, rebasing and how a three-way merge finds conflicts.40 min
  35. Writing for engineersREADMEs, design docs, commit messages and bug reports people can act on.35 min
  36. Scoping and estimating workA vague request cut into tickets with acceptance criteria and a first shippable slice, estimates as ranges, an honest update for a slip, and a ticket linked to its pull request.30 min
  37. Roles, résumé and the searchWhat AI, ML and data engineering postings ask for, résumé bullets that carry a number, a 100-word project summary, a take-home as a portfolio piece and how to read an offer.35 min
  38. Interviews and your portfolioShow your work, and solve interview problems the way professionals do.40 min
  39. Code reviewWhat to look for, how to give and take feedback, and automating the routine checks.35 min
  40. Reading unfamiliar codeFind your way around a big codebase fast, with tools that map it for you.35 min
  41. Working with AI coding toolsGive agents good context, keep tasks small, and check every change they make.35 min
  42. Capstone: ship a production AI serviceAn AI-backed service with validation, caching, rate limits, fallbacks, logging and evals.45 min
  43. Capstone: take a model to productionA churn model through a data contract, a tracked run, a registry gate by slice, hybrid serving, a PSI monitor and one retrain, shipped with a datasheet, a model card, a fairness audit, a design doc and a runbook.45 min
Part of a free course 11 lessons

Free MLOps course: running machine learning in production

Take a model from a notebook to a system that stays right: track and reproduce every run, search settings honestly, version data and models, promote through gates, keep training and serving in step, choose batch or online, catch drift, and retrain through a pipeline. Every number is measured by the lesson's own code.

Never trained an AI? Teach a small one to write, right in your browser. It takes about 20 seconds and needs no account. Teach it to write